Disentangling Style and Content in Anime Illustrations
Authors: Sitao Xiang
Hao Li
Venue:  (2020)
Keywords: Adversarial Training
Generative Models
Style Transfer
Anime

Abstract: Existing methods for AI-generated artworks still struggle with generating high-quality stylized content, where high-level semantics are preserved, or separating fine-grained styles from various artists. We propose a novel Generative Adversarial Disentanglement Network which can disentangle two complementary factors of variations when only one of them is labelled in general, and fully decompose complex anime illustrations into style and content in particular. Training such model is challenging, since given a style, various content data may exist but not the other way round. Our approach is divided into two stages, one that encodes an input image into a style independent content, and one based on a dual-conditional generator. We demonstrate the ability to generate high-fidelity anime portraits with a fixed content and a large variety of styles from over a thousand artists, and vice versa, using a single end-to-end network and with applications in style transfer. We show this unique capability as well as superior output to the current state-of-the-art.

Under review as a conference paper at ICLR 2020
DISENTANGLING STYLE AND CONTENT IN ANIME ILLUSTRATIONS
A Anonymous authors Paper under double-blind review
ABSTRACT
Existing methods for Al-generated artworks still struggle with generating high-quality stylized content, where high-level semantics are preserved, or separating fine-grained styles from various artists. We propose a novel Generative Adversarial Disentanglement Network which can disentangle two complementary factors of variations when only one of them is labelled in general, and fully decompose complex anime illustrations into style and content in particular. Training such model challenging, since given a style, various content data may exist but not the other way round. Our approach is divided into two stages, one that encodes an input image into a style independent content, and one based We demonstrate the ability to generate high-fidelity anime portraits with a fixed content and a large variety of styles from over thousand artists, and vice versa, using a single end-to-end network and with applications in style transfer. We show this unique capability as well superior output to the current state-of-the-art.
dual-conditional generator.
INTRODUCTION
Computer generated art (Hertzmann 2018) has become topic of focus lately, due to revolutionary advancements deep learning. Neural style transfer (Gatys al. 2016) is a groundbreaking approach where high-level styles from artwork can be re-targeled to photographs using deep neural networks. While there has been numerous works and extensions on this topic, there are deficiencies in existing methods. For complex artworks, the methods that rely on matching neural network features and feature statistics, do not sufficiently capture the concept of style the semantic level. Methods based on image-to- image translation [Isola et al. 2017) are able to learn domain specific definitions of style, but do not scale well large number of styles.
In addressing these challenges, we found that style transfer can be formulated as a particular instance of general problem, where the dalaset has complementary factors of variation, with one of the factors labelled, and the goal is to train a generative network where the two factors can be fully disentangled and controlled independently. For the style transfer problem, we have labelled style and unlabelled content.
Based on various adversarial training techniques, we propose solution to the problem and call our method Generative Adversarial Disentangling Network. Our approach consists of two main stages. First, we train a indeeenndeenn content encoder, then we introduce a dual-conditional generator based on auxiliary classifier GANs.
We demonstrate the disentanglement performance of our approach on a large dataset of anime portraits with over a thousand artist-specific styles, where our decomposition approach outperforms existing methods in terms of level of details and visual quality. Our method can faithfully generate portraits with proper style-specific shapes and appearances of facial features, inchuding eyes, mouth, chin, hair, blushes, highlights, contours, as well as overall color saturation and contrast. To show the generality of our method, we also include results on the NIST handwritten digit dataset where we can disentangle between writer identity and digit class when only the writer is labelled, alternatively when only the digit is labelled.

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND
Neural Style Transfer. Gatys et al. Gatys et al. (2016) proposed a powerful idea of decomposing an image into content and style using deep neural network. By matching the features extracted from one imput image using a pre-trained network and the Gram matrix of features of another image, one can optimize for an output image that combines the content of the first input and the style of the second one. One area of improvement has been to first identify the best location for style source in the style image for every location in the content image. Luan et al. Luan et al. (2017) use masks from either user imput or semantic segmentation for guidance. Liao et al. Liao et al. (2017) extends this approach by finding dense correspondences between the images using deep neural network feat in a coarse-to- to-fin fashion. In another line of research, different representations for style have been proposed. In Huang et al. Huang and Belongie (2017), the style is represented by affinc transformation paramcters of instance normalization layers. These methods go slightly beyond transferring texture statistics. While we agree that texture features are an important part of style, these texture statistics do not capture high-level style semantics. Our observation is that, the concept of "style" is inherently domain-deeenceen,, and no set of features ures defined a priori can adequately handle style transfer problem in all domains al once, let alone mere texture statistics. In our views, a better way of addressing style transfer would be to pose it as an image-to-ing to-image translation problem.
Image-to-image Translation. Isola et al. introduced the seminal work on image-to- image translation in [Isola et al.) 2017) where extensive source and target training data need to be supplied. Several extensions to this approach, such as CycleGAN (Zhu et al. 2017) and DualGAN (Yi et al. 2017), removed the need for supervised training, which significantly increases the applicability of such methods. Of particular interest, Zhu et al. Zhu et al. (2017) demonstrates the ability to translate between photorealistic images and the styles of Van Gogh, Monet and Ukiyo-e. While the results are impressive, different network is still required for each pair of domains of interest. One solution to this, is to train an encoder and generator for each domain such that each domain can be encoded to and generated from a shared code space, as described in Liu et al. Liu et al. (2017). We wish to take one step further and use only one set of networks for many domains: the same encoder encodes each domain into common code space, and the generator receives domain labels that can generate images for different domains. In a way, rather than considering each style as separate domain, we can consider the whole set of images as single large domain, with content and style being two different factors of variation, and our goal is to train encoder-decoder that can disentangle these two factors. The idea of supplying domain information so that a single generator can generate images in different domains is also used in StarGAN (Choi et al. 2018). But in their work, there is explicit content space, so it does not achieve the goal of disentangling style and content. Our view of style transfer being an instance of disentangled representation is also shared by (Huang et al. 2018), but their work considers mapping between two domains only.
Disentangled Representation. DC-IGN (Kulkarni et al. 2015 achieves clean disentan- glement of different factor of variation in a set of images. However, the method requires very well-structured data. In particular, the method requires batches of training data with the same content but different style, as well as data with the same style but different content. For style transfer, it is often impossible to find many even two images that depict the exact same content in different styles. On the other extreme, the method of Chen et al. Chen et al. (2016) is unsupervised and can discover disentangled factors of variations from unorganized data. Being unsupervised, there is no way to enforce the separation of a specific set of factors explicitly. The problem we are facing is in between, as we would like to enforce the meaning of the disentangled factors (style and content), but only one of the factors is controlled in the training data, as we can find images presumed to be in the same style that depict different content, but not vice versa. (Mathieu et al. 2016) is one example where the setting is the same to ours. Interestingly, related techniques can also be found in the field of audio processing. The problem of voice conversion, where audio speech is mapped to the voice of a different speaker, has a similar structure to our problem. In particular, our approach is similar to (Chou et al. 2018
2

---

Under review as a conference paper at ICLR 2020
3 METHOD
The training data must be organized by style. However, fine-graine grained labels of styles are difficult to obtain. Hence, we use the identity of artists, as proxy for style. While an artist might have several styles and it might evolve over time, using an artist's identity as a proxy for style is a good approximation and an efficient choice, since the artist's label are readily available. As in (Chou al. 2018), our method divided into two stages.
Stage 1: Style Independent Content Encoding. In this stage, the goal is to train an encoder that encode as much information as possible about the content of an image, but no information about its style. We use per pixel L2 distance (not its square) for the reconstruction loss: for two (3-channel) images X, Y Rhxwx3, by || Yil without any subscript we mcan this distance:
h
W
| YII
ΣΣ hw
|x,,
Yij
We start from solution that did not work well. Consider simple encoder-decoder network with encoder E-) and decoder G(-) whose sole purpose is to minimize the reconstruction loss:
Lree = E 11= E
- G(E(z))I
min Cree E,G
x->((a)
where p(z) is the distribution of training samples. To prevent The encoder from encoding style information, we add an adversarial classifier C(-) that tries to classify the encoder's ' output by artist, while the encoder tries to maximize the classifier's loss:
Lc =
E z.a=p(z.a)
[NL((C((())))))
min Lc
min Lree - AAc E,G
(1)
where a is the integer ground truth label representing the author of image x and p(x, a) is the joint distribution of images and their authors. À is weight factor. To resolve the conflicting goal that the generator needs style information to reconstruct the imput but the encoder must not encode style information to avoid being successfully classified, we can learn a vector for cach artist, representing their style. This "style code" is then provided to G(+.)) which now takes two inputs. We introduce the style function S(.) which maps an artist to their style veclor. Now the objective is
Lree E a.0=p(a.a)
[| - G(E(()),((a)))
min Croc ALc E,G.S
Note that S(.) is not another encoding network. does not see the inpul image, but only its artist label. This is essentially identical to the first stage of (Chou et al. 2018). Tn our experiments, we found that this method does not adequately prevent the encoder from encoding style information. We discuss this issue in appendix C
We propose the following changes: instead the code E(x), C(-) tries to classify the generator's output ((E(z),S(aa)) which is the combination of the content of and the style of a different artist a'. In addition, akin to Variational Autoencoders (Kingma and Welling 2013), the output of E(-) and 5(.) are parameters of multivariate normal distributions and we introduce KT.-divergence loss to constrain these distributions. To avoid the equations becoming too cumbersome, we overload the notation bit so that when E(-) and S(-) are given to another network as input we implicitly sample a code from the distribution. The optimization objectives becomes
Le
E (LLL(C(G(E(),SS()))).... x.a-p(a.a] a' d'mp(a)
min Lo
Lree=
E
[| x| -EE(((E)),(()II
LE-KL = E DKL (E()) x>p(())
Ls.KL E (DL I))] a>pla)
min free dcLo + NE + L...LL E,G,S
3

---

Under review as a conference paper at ICLR 2020
Dual- Cnal-Conditional Generator. It is well known that autoencoders typically produce blurry outputs and fail to capture texture information which is an important part base approach on auxiliary classifier GANs (Odena
Stage 2:
of style. To condition styles, eet al. 2017). First, as usual, a discriminator (-)) tries to distinguish wetw real and generated samples, but instead of binary cross entropy, we follow (Mao al. 2017 and least squares loss. The discriminator's 's loss is
D.rreal
E (D(x) 1) z>p(a)
D-ffake
EE (D()) +
xmp(a) a'mp(a)
while the generator's loss against the discriminator is
LD-adv
E xo-p(a)) a'<p(a)
[D(GLE())
Here the encoder E(.), generator G((:.) and style function S(.) are inherited from stage Note that in all the equations a is sampled jointly with æ but a' is independent. Then, similar to (Odena et al. 2017), a classifier is trained to classify training images to their correct authors:
LC 2,0-p(z.a)
E (NLL(C2(#),a))
Unlike the generator and encoder, C2(-) is a different classifier than the one in stage 1, thus we add subscript lo disambiguale, and shall refer to the stage 1 classifier as Ci(-). The generator try to generate samples that would be classified as the artist that it is conditioned on, by adding this to its loss function:
LCr-adv E [NUL(CC(G(EE)))5(d)))).
(2)
2~<((2) d'~p(a)
But we differ from previous works on conditional GANs in the treatment of generated samples by the classifier. In (Odena et al.) 2017) the classifier is cooperative in the sense that would also try minimize equation
In other works like (Choi al. [2088) Chou let al. 2018 Mathieu et al. (016)) there is no special treatment for generated images, and the
classifier does not oplimiie any loss function on them. In (Springenberg 2013) the classifier is trained to be uncertain about the class of generated samples, by maximizing the entropy of C((C())))
We train the classifier it to explicitly classify generated images conditioned on the style of as "not a". For this, we define what we call the "negative log-unlikelihood:
NLU(y, i) - log(1 [1 y/)
and take this as the classifier's loss on generated samples:
LCy-lake
(NI(CC((GEEE)))S)))))))) d'>p(a)
E
2>p(())
We discuss the effect of an adversarial classifier in appendix C While the discriminator and the classifier are commonly implemented as a single network with two outputs, we use separate networks for C2(+) and D(-). To enforce the condition on content, we simply take E and require that the generated samples be encoded back to its content input:
Loont
(I)) Ex)
x>p(())
a' a'mp(()
And our training objective for this stage is:
min L + L
min LCg-real + LCg--ake C2
D
min ApLp-adv + Ac, LCr-ady + Acont Ccont + As-KL G.S
Note that E(.) is fixed in stage 2. Since at the equilibrium C2(() is expected to classify real samples effectively, it would be good to pre-train C2(.) on real samples only prior to stage 2. The training procedures summarized in figure KT-divergence omitted for clarity.
4

---

Under review as a conference paper at ICLR 2020
Loves
Lersin Loxso
10
p(x.a), d' <p(())
Elid)
3,0 me p(t,))
~ pi)
(a) Stage
(b) Stage
Figure 1: Training procedure. Squares are networks and rounded rectangles are loss terms. Blue parts are for E, S and G, red parts are for D and C. Black parts are common to both.
Table Common part of network architecture
Layer
SC
SC
SC
SC
SC
SC
F
Channel
3
32
64
128
256
512
1024
2048
Size
256
128
64
32
16
4
EXPERIMENTS
For our main experiments presented here, we use anime illustrations obtained from Danboort Furher experiments the NIST handwritten digit dataset can be found in appendix B in which we also provide visualization and analysis of the encoder's output distribution for a better assessment of our method.
Dataset. We took all images with exactly artist tag, and processed them with open source tool AnimeFace 2009 (nagadomi, 2017] to detect faces. Fach detected face is rotated into upright position, then cropped out and scaled to 256 x 256 pixels. Every artist who has at least 50 image were used for training. Our final training set includes 106,814 images from 1139 artists.
Network Architecture. All networks are built from residue blocks (He al. 2016). While the residue branch is commonly composed of two convolution layers, we only use one convolution per block and increase the number of blocks instcad. ReLU activations are applicd on the residue branch before it is added to the shorteut branch. For simplicity, all of our networks had an almost identical structure, with only minor differences in the /uput/oulput layers. The common part is given in table [ The network is composed of the stride 2 convolution,
sequence of blocks in the first row (C stride convolution, fully connected) with the number of output channels in the second row and spatial size of output feature map in the third row. For the generator, the sequence runs from right to left, while for other networks the sequence runs from left to right. On top of this common part, fully connected /oupput layers are added for each network, with appropriate number of imput/ output features: for classifiers C.(+) and C2()), the number of artists is 1,139; for the discriminator D(-), we use 1; for the encoder E(-), we use parallel layers with 256 features for mean and standard deviation of output distribution; for the generator G(:,-), the sum of the content and style dimensions is 512.
Training. We weight the loss terms with the hyperparameters in table 2 which also inchudes training parameters. S(-) treated differently, since is not network but just matrices storing style codes. We use the same learning rate for every network in single stage. In stage 2, we use RMSprop for our weight updating algorithm since using the
ddanbooru.dommai.n

---

Under review as a conference paper at ICLR 2020
Table 2: Weighting and training hyperparameters
Weight
Value
Learning rate
Aa1
0.2
Stage
Algorithm
Batch
Time
S
Others
λÆ-KL
10-4
Ag-KL
10-5 1
0.005
5 10-5
Adam
400k
AD
C2 pro-train
10
Adam
16
200k
ACa
0.01
2 x 10-3
RMSprop
400k
Acont
0.05
Figure 2: Images generated by fixing the style in each group of two rows and varying the content. Two different styles shown. Leftmost column taken from training set, courtesy of respective artists. Top group: Sayori. Bottom group: Swordsouls.
momentum would sometimes cause instabilities in GANs. In every other stage the Adam optimizer. Timing is measured as number of iterations.
RESULTS
Disentangled Representation of Style and Content. To test our main goal of dis- entangling style and content. in the generator's input code, show that the style code and content code do indeed only control the respective aspect of the generated image, by fixing one and changing the other. In figure 2 in each group of two rows, the leftmost two images are examples of illustrations drawn by artist in the datasct, and to the right are 12 samples generated from the style the artist. Since this can be expected from conventional dlas-conditionall GAN, it is not the main contribution of our method. In particular, the strength our method, is the ability to generate the same content. different styles where facial shapes, appearances, and aspect ratios are faithfully captured. We refer to Appendix [ for additional results and discussions. Figure shows 35 images generated with a fixed content and different style codes.
Style Transfer. As a straightforwart application, we show some style transfer results in figure with comparisons to existing methods, the original neural style transfer (Gatys et 2016) and StarGAN (Choi et al. 2018). We can see that neural style transfer seems to mostly apply the color of the style image to the content image, which, by definition, not only fails to capture the style of the style image but also alters the content of the content image. StarGAN managed to transfer the overall use of color of the target artist and some other prominent features like size of the eyes, but otherwise fails to capture intricate style elements. Our method transfers the style of the target artist much more faithfully.

---

Under review as a conference paper at ICLR 2020
Figure Images generated from a single content code and an assortment styles. Including both style of artists from the training set and style codes randomly samples from the style distribution
Evaluation. Other than visual results, there is no well-established quantitative measure for the quality of style transfer methods and accessing experts for anime styles is challenging for a proper user study. The evaluation in (Chou et al. 2018) is audio specific. In (Choi ltt al., 2018), the classification accuracy on the generated samples is given as quality mcasure. However, this seemingly reasonable mcasure, as we shall argue in appendix C.2 is
not adequate. Nevertheless, we do report that the samples generated by our generator can be classified by style by our classifier with 86.65% top-1 accuracy. Detailed testing procedure, results and analysis can be found in appendix C.2 A comprehensive ablation study is given in Appendix C where major differences from previous approaches are evaluated. The effectiveness of stage 1, the disentangling encoder, is evaluated quantitatively in section [.4]
Limitations. Due to the relative scarcity of large labelled collection of artworks depicting consist set of subjects, we have thus far not collected and tested our method on examples beyond portraits and artistic styles beyond anime illustrations. We also noticed some inconsistencies in small features, such as eye colors, presumably due to small features being disfavored by the per-pixel reconstruction loss in stage 1, as well as our choice of architecture with fixed-size code rather than fully convolutional. Additional discussions can be found in Appendix A
CONCLUSION
We introdeeed Generative Adversarial Disentangling Network which enables true semantic- level artwork synthesis using a single generator. Our evaluations and ablation study indicate that style and content can be disentangled effectively through our two-stage framework, where first a style independent content encoder is trained and then, a content and style- conditional GANs is used for synthesis. While we believe that our approach can be extended to a wider range of artistic styles, we have validated our technique on various styles within the context of anime illustrations. In particular, this techniques is applicable, as long as we disentangle two factors of variation in dataset and only one of the factors is labelled and controlled. Compared to cxisting methods for style transfer, we show significant improvements in terms of modeling high-level artistic semantics and visual quality. In the future, hope to extend our method to styles beyond anime artworks, and we are also interested in learning to model entire character bodies, even entire scenes.
7

---

Under review as a conference paper at ICLR 2020
Figure 4: Style transfer results. In the top two rows, in each column are two samples from the training dataset by the same artist. In each subsequent group of three rows, the leftmost image is from the training dataset. The images to the right are style transfer results generated by three different methods, from the content of the left image in the group and from the style of the top artist in the column. In each group, first row is our method, second row StarGAN and third row is neural style. For neural style, the style image is the topmost image in the column. Training samples courtesy of respective artists. Style samples, from left: Sayori, Idoolo, Peko, Yabuki Kentarou, Coffec-kizoku, Mishima Kurone, Ragho no Erika. Content samples, from top: Kantoku, Koi, Horiguchi Yukiko.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Πya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172 21800, 2016.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image transla- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8789 8797, 2018.
Ju-chieh Chou, Cheng-chieh Yeh, Hung-yi Lee, and Lin-shan Lee. Multi-target voice conver- sion without parallel data by adversarially learning disentangled audio representations. Xrxi preprint ar arXix:184.02811 2018.
Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems, pages 262-270, 2015.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414-2423, 2016.
Leon Gatys, Alexander $ Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. Controlling perceptual factors in ncural style transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Aaron Hertzmann. Can computers create art? CoRR, abs/ 1801.04486, 2018. URL http:://rxiv.orggabb////e 044866
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 1510 1519. IEEE, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image- to-image translation. In Proceedings of the European Conference Computer Vision (ECCV), pages 172-189, 2018.
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In 2017 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967 5976. IEEE, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812, 04948, 2018.
Hadi Kazemi, Seyed Mehdi Iranmanesh, and Nasser Nasrabadi. Style and content disen- tanglement in generative adversarial networks. In 2019 IEFE Winter Conference on Applications of Computer Vision (WACV), pages 848 848-856. IEEE, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arxis preprint arX1::1812.6114, 2013.
Tejas D Kulkarni, William Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphies network. In Advances in neural information processing systems, pages
2015.
Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep image analogy. ACM Transactions Graphics (TOG), 36(4):120, 2017.

---

Under review as a conference paper at ICLR 2020
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems, pages 700-708, 2017.
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In 2017 IEEE Conference on Compuler Vision and Pattern Recogrition (CVPR), pages 6997 7005. IEEE, 2017.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2813 2821. IEEE, 2017.
Michael F Mathicu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Discntangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, pages 5040-5048, 2016.
nagadomi. Animeface 2009. http:s//gghhh...cco////ag/d/aaae 2009 2017. Ac- cessed: 2018-1-9.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International Conference on Machine Learning, pages 2642 2651, 2017.
Oren Rippel, Michael Gelbart, and Ryan Adams. Learning ordered representations with nested dropout. In International Conference on Machine Learning, pages 1746-1754, 2014.
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint orXiw:1511.0633 1663999011.0..1111 2015.
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. Technical report, 10x1::1905.10499, may 2019. URL https://arxiv org/ arrg/bbs/1900 10498
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages
IEEE, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 JEEE International Conference on, pages 2242-2251. TEEE, 2017.
10

---

Under review as a conference paper at ICLR 2020
A ADDITIONAL DISCUSSIONS
A.1 THOUGITT ON STYLE AND STYLE TRANSFER
While works on this topic has been numerous, we feel that one fundamental question is not often carefully addressed: what is style, in a deep learning setting?
As stated in (Gatys et al. 2016), which is based on an earlier work on neural texture synthesis [Gatys et al. 2015), the justification for using Gram matrices of neural network features as representation of style is that captures statistical texture information. So, in essence, "style" defined as such is a term for "texture statistics", and the style transfer is limited to texture statistics transfer. Admittedly, it does it smart ways, as a sense the content features are implicitly used for selecting which part of the style image to copy the texture statistics from.
As discussed in section 2 above, we feel that there is more about style than just feature statistics. Consider for example the case of caricatures. The most important aspects of the style would be what facial features of the subjects are exaggerated and how they are exaggerated. Since these deformations could span a long spatial distance, they cannot be captured by local lexture statistics alone.
Another problem is domain dependency. Consider the problem of transferring or preserving color in style transfer. If we have landscape photograph taken during the day and want to change it to night by transferring the style from another photo taken during the night, or if we want change the season from spring to autumn, then color would be part of the style we want to transfer. But if we have still photograph and want to make it an oil painting, then color part of the content, we may want only the quality of the strokes of the artwork but keep the colors of our original photo.
People are aware of this problem and in (Gatys al.) 2017 : two methods, luminance-only style transfer and color histogram matching, are developed to optionally keep the color of the content image. However, color is only one aspect of the image for which counting it as style vs. content could be ambiguity. For more complicated aspects, the option to keep or to transfer may not be easily available.
We make two observations here. First, style must be more than just feature statistics. Second, the concept of "style" is inherently domain-dependent. In our opinion, "style" means different ways of presenting the same subject. In each different domain, the set of possible subjects is different and so is the set of possible ways to present them.
So, we think that any successful style transfer method must be adaptive to the intended domain and the training procodure must actively use labelled style information. Simple feature based methods will never work in the general setting. This includes previous approaches which explicitly claimed to disentangle style and content, such as in (Kazemi al. 2019) which adopts the method in the original neural style transfer for style and content losses, and also some highly accomplished methods like (Liao et al. 2017).
As a side note, for these reasons we feel that some previous methods made questionable claims about style. In particular, works like (Huang et al. 2018) and StyleGAN (Karras let al.] 2018) made reference to style while only being experimented on collections of real photographs. By our definition, in such datasct, without a careful definition and justification there is only one possibly style, that is, photorealistic, so the distinction between style and content does not make sense, and calling certain subset of factors "content" and others "style" could be an arbitrary decision.
This is also why we elect to not test our method on more established GAN datasets like CelebA or LSUN, which are mostly collections of real photos.
A.2 MORE ON RESULTS
The differences in style can be subtle, and readers may not be familiar enough with anime illustrraiion to be able to recognize them. We hint at several aspects to which the reader
11

---

Under review as a conference paper at ICLR 2020
may want to pay attention: overall saturation and contrast, prominence of border lines, overall method of shading (flat vs. 3-dimensional), size and aspect ratio of the eyes, shape of the irides (round vs. angular), shininess of the irides, curvature of the upper eyelids, height of lateral angle of the eyes, aspect ratio of the face, shape of the chin (round vs. angular), amount of blush, gramularity of hair strands, prominence of hair shadow on forehead, treatment of hair highlight (intensity, no specularities / dots along a line / thin line / narrow band/ wide band, clear smooth boundary / jagged boundary / fuzzy boundary, etc.).
If we specifically examine these style elements in the style transfer results in figure | it should be evident that our method has done a much better job. That being said, our network is not especially good at preserving fine content details.
This can be said to be a result from our choice of architecture: we use an encoder-decoder with a fixed-length content code while StarGAN uses a fully convolutional generator with only down-sampling layers, which is able to preserve much richer information about the content image.
Part of the reason is that we consider it a desirable feature to be able to sample from the content distribution. In convolutional feature maps, features at different locations are typically correlated in complicated ways which makes sampling from the content distribution impractical. Unsurprisingly previous works adopting fully convolutional networks did not demonstrate random sampling of content, which we did.
Another reason is that we feel that the ability of fully convolutional networks to preserve content has a downside: it seems to have a tendency to preserve image structures down to the pixel level. As can be observed in figure [ StarGAN almost exactly preserves location of border lines between different color regions. As we have mentioned, part of the artists' style is different shape of facial features, and a fully convolutional network struggle to capture this. We suspect that for a fully convolutional architecture to work properly, some form of non-rigid spatial transformation might be necessary. This can be one of our future directions.
A.3 INCONSISTENCY OF SMALL FEATURES
As can be seen from many of the visualizations, the color of the eyes, and facial expression which largely depends on the shape of the mouth changing the style codes. We thought that this could be the result of the choice of the reconstruction loss function in stage 1. The loss is averaged across all image pixels, which mcans the importance of a feature to the network is proportional to its area. With the code length being limited, the network prioritizes encoding large features, which can be seen from the fact that large color regions in the background are more consistent than eyes and mouth.
sometimes not well preserved when
For this particular problem, we can utilize additional tags: almost all images gathered from Danbooru are tagged according to eye color and facial expressions. We could train the encoder classify for these tags while the generator could be conditioned on these tags. For general problems however, we may need loss functions that are more aligned with humans' perception of visual importance.
Another observation is that sometimes in generated image the two eyes are of different color. We point out that the dataset do contain such images, and while still rare, the prevalence of heterochromia in anime characters is much higher than in real life. We think that this could be enough to force the encoder to use two different set of dimensions to encode the color of each eye, thus causing random samples to have different colored eyes.
B EXPERIMENTS ON NIST DATASET
We conduct further experiments the recently released full NIST handwritten digit dataset Yadav and Bottou, 2019), which is a superset of the MNIST dataset. In addition to having many more imgess (a total of 402,953 images), the NIST dataset comes with more metadata, among which the identity of the writer in particular is useful for our study (3579 different writers). We show the ability of our network to disentangle between digit class and writer
12

---

Under review as a conference paper at ICLR 2020
Table 3: Weighting and training hyperparameters for NIST
Parameter
W vs. D + R
D vs. W
+
R
Aci
0.1
0.1
λE-KL
10-4
10-4
AS-KL
10-4
10-4
AD
1
1
Aça
0.2
1
Acent
0.5
0.1
Stage time
300k
300k
C2 pre-train time
800k
100k
Stage 2 time
320k
320k
identity when only the writer label is known but not the digit label, or vice versa, when only the digit label is known but not the writer label.
Strictly speaking, we are not precisely disentangling between digit class and writer identity: as we will show shortly, non-negligible variation exists when the same person is writing the same digit, so there are actually three set of factors that determines the appearance of a written digit: digit class (denoted D), writer identity (W) and the rest (R). We have labels for D and W in the dataset. Our purpose is to disentange between labelled variations and unlabelled variations. So, with the label for digit and writer available, we show the following
two experiments: disentangling between 2 and W R when only digit label is used for training, and disentangling between W and D + R when only writer label is used for training.
B.1 SETUP
The images of handwritten digits are of size 28 x 28. Accordingly, the size of our network was reduced to 3 levels with two convolution residue blocks each, with 64, 128 and 256 channels.
The code length for digit class, writer identity and the rest of features 32 respectively. Thus, for W vs. D + R the labelled feature had 128 dimensions and the unlabelled feature had 16 32 = 48 dimensions, while for D vs. W + R the labelled feature had 16 dimensions and the unlabelled feature had 128 + 32 = 160 dimensions.
16, 128 and
Weighting parameters and training times also adjusted and given in table
B.2 DISENTANGLING BY WRITER LABEL
First we show the scenario where the label for writer identity is known, similar to the main experiment where the artist is known. Generated samples of all digits and randomly selected writers are shown in figure 5 Samples in the same column is from the same writer, whose code is that learned by S(-). while samples in the same row has the same + R code, which is obtained by taking the average of F(x)) from all images in the same digit class.
Although we fell that is clear that we can control style and content independently, given the fact that the perception of style is subjective, here we demonstrate how cleanly W has been disentangled from + R by visualizing the output distribution of E(-).
Principal Component Analysis is commonly used for projecting high dimensional feature space to a low dimensional one. But considering that feature scaling can easily skew the result, and because in section C.1 we will be visualizing the change of the output distribution of E(-) during training which require the same projection be used for different distributions, we chose to use Nested Dropout (Rippel et al. 2014) that code dimensions automatically learned and ordered by importance and we can simply project to the first two dimensions for visualization.
Figure 6a shows the distribution of E(x) over the whole dataset projected to the first two dimensions, with color representing digit class. In figure 6b is the output of vanilla VAE
13

---

Under review as a conference paper at ICLR 2020
0
0
0
0
0
/
/
1
1
/
/
/
2
2
2
2
a
2
a
2
2
2
2
3
2
2
2
2
2
2
2
2
2 3
2 3 4 5 6
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
Y
Y
4
4
4
4
4
4
4
4
4 5 4 7
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
b
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 8 9
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
7
9
9
7
9
8
9
9
9
0
9
9
Figure 5: Generated samples when disentangling W from D + R
(a) Stage encoder for W vs. D+ R
(b) VAE encoder
Figure 6: Comparison of output distribution between stage 1 encoder and vanilla VAE.
with the same structure trained on the same dataset. Comparing vanilla VAE, E(-) produced a distribution in which images depicting the same digit are more closely clustered together while images depicting different digits are more cleanly separated. In particular, we obtained 10 clcarly distinct clusters with only two dimensional feature space, while with VAE three of the digits are almost completely confused (they needed third dimension to be separated). This is because we have purged information about the writer from the encoder: Since the identity of the writer independent from digit class, by removing writer information, variation within the same digit class were reduced. Remember that we are able to achieve this clustering effect without knowing digit label during training.
B.3 DISENTANGLING BY DIGIT LABEL
Our method is not limited the case where the labelled feature correspond style while the unlabelled feature correspond to content. In fact, as we will show here, we can achieve the direct opposite, with the labelled feature corresponding to content and the unlabelled feature corresponding to style: we disentangle D from W + R. Generated samples of all digits and randomly selected writers are shown in figure 7 Samples the same row represents the same digit, whose code is that learned by S(.), while samples in the same column has the same W + R code, which obtained by taking the average E(x) from all images r by the same writer.
14

---

Under review as a conference paper at ICLR 2020
0 0
0
0
1 1 /
1
1
1
/
1
/
/
1
2
2
2
Z
2 2
2
2
2
2.
2
2
3
2
2
2
2
2
a
2
2
2
3 3 3 3
3
3 3
3
3
3
3
3
3
3
3
3 3 4
3
3
3
3
3
3
3
3
4 4 4 y
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4 5 6
4
4
J 5
5
r
5
5
5
S
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6 7 8 9
7 7
7
7
1
7
2
7
7
7
7
7
7
7
7
7 7
7
7
7
7
7
7 8
7 8 9
y
8
8
8
i
8
8
8
8
&
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
Figure 7: Generated samples when disentangling D from W + R
0
0
0
0
0
0
1
/
1
1
1
1
1
1
1
1
2
2
2 3 4
2 3 4
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3 4 5 6
4
4
4
4
4
4
4
4
4
4
4
5
5
55
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
7
1777
7
7
7
7
7
7
7
7
7
7
7
8
0
8
8 8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
Figure 8: Variation within the same digit written by the same writer
We can observe that the variation in the same row is much more dramatic in figure than in figure 5 despite that in both cases can (roughly) consider each row to have the same content and each column to have the same style. This is in fact the correct behaviour: in figure 5 the variation in each row is due W only while in figure the variation in each row due to the combined effect W and R. This reveals the fact that, even within multiple images of the same digit written by the same person, the variation can be significant.
We can verify this in figure g in the left 5 columns are samples by the same writer taken from the training dataset. In the right 10 columns are samples generated by the generator in section B.2 The writer code is that of the same writer as the samples to the left, as learned by S(-), while the D + R codes are obtained by adding some noise to the average E(x) of different digit classes. The training data shows certain amount of variation within the same digit written by the same writer, and we can achieve similar variations by fixing W and D and only changing R.
Similar to section B.2 we look at the distribution of E(x) again. This time, since the digit class is the labeled feature which want to purge from the encoder, if we were successful, the distributions of each digit should be indistinguishable The distribution of each individual digit are given in figure These distributions are indeed very similar. Compare this to figure 6b
B.4 QUANTITATIVE ANALYSIS OF DISENTANGLING ON NIST DATASET
gave qualitative evaluation of the full pipeline by various visualizations. In this section we focus on the stage 1 encoder and give some quantitative evaluation of the effect of disentanglement.
In sections B.2 and B.3
15

---

Under review as a conference paper at ICLR 2020
Figure 9: Distribution of each digit from stage encoder for D vs. W + R
Table 4: Mean Euclidean distance of a sample to the center of its class
Encoder
By writer
By digit
Whole dataset
Ew
1.2487
0.2788
1.2505
ED 0.7929
1.2558
1.2597
Ey
1.2185
0.4672
1.2475
(a) First 2 dimensions
Encoder
By writer
By digit
Whole dataset
Ew
2.6757
2.0670
2.6957
ED
2.4020
2.6699
2.7409
Ev
2.6377
1.7629
2.7363
(b) First 8 dimensions
The encoder aims to encode the unlabelled feature in the dataset while avoiding encoding the labelled feature. Intuitively, this means that each unlabelled class should be distributed differently the encoder's 's code space while each labelled class should be distributed similarly. We make this intuition precise by considering two metrics: average Euclidean distance of sample in the code space to the mean of its class, and the performance of a naive Bayesian classifier using the code as feature.
First, consider the average distance metric. For the disentangling encoder trained on writer label (denoted Ew in this section), if it does successfully encode digit information but not writer information, samples of the same digit should be clustered more tightly together than the whole dataset, while samples by the same writer should not be clustered noticeably more tightly than the whole dataset. In addition, samples of the same digit should be clustered more tightly together than for VAE encoder not trained to disentangle (Ev), and samples by the same writer should be clustered less tightly. For the disentangling encoder trained on digit label (Ep), the opposite should be true. So, for each of Ew, Ey and Ep, we compute the average Fuclidean distance of sample to the center of its class, with the classification being by writer, by digit, or just. the whole as single class.
We fond that the most distinctive information are all encodes in the first few dimensions of the code space, with later dimensions being more random, thus disturbing the clustering effect rather than helping. So we truncate the encoder's output to either two dimension or cight dimcnsions, and give the result for both. The result is shown in table 4 In each row, the red number should be noticeably smaller than the black mumber while the blue number should be similar to the black number. In each column, the red number should be noticeably smaller than the black number, while the blue number would preferably be larger.
16

---

along with ablation studies to Under review as a conference paper at ICLR 2020
Table 5: Average probability given to the correct class
Encoder
By writer
By digit
Encoder
By writer
By digit
Ew
0.000293
0.9001
Ew
0.000363
0.9327
ED 0.001441
0.1038
ED
0.002845
0.1015
Ev
0.000337
0.6179
Ey
0.000843
0.9380
(a) First dimensions
(b) First 8 dimensions
Table 6: Average rank of the correct class
Encoder
By writer
By digit
Encoder
By writer
By digit.
Ew
1608
1.12
Fw
1330
1.12
ED 582
5.20
ED
422
3.98
Ev
1409
1.49
Ey
838
1.08
(a) First dimensions
(b) First 8 dimensions
We can see that Ew causes the samples to cluster very tightly by digit while EDD has the opposite effect, which is consistent with the visualization in figures Gaa and g Conversely, ED causes the sample to cluster somewhat more tightly by writer, while Ey has the opposite effect.
Second, we study the classification performance in the encoder's output space. For this purpose; we a naive Bayesian classifier with each class modelled by axis-aligned multivariate normal distribution. That is, for each class we take all samples of that class, fit normal distribution with the covariance matrix being a diagonal matrix, then for each sample compute the likelihood of it being generated by each class and give a class distribution by Bayes' thcorem. For each of Ew, Ey and ED we classify by writer and by digit, with first two dimensions and then with first 8 dimensions of the code space.
The average probability assigned to the correct class, average rank of the correct class and lop-1 classification accuracy are given in tables 5 i and 7 respectivvly. Similar to the average distance metric, each lable the red number should be better than the black number the same column while the blue number should be worse, with "better" meaning high average probability to the correct class, lower average rank and higher top-1 accuracy. A lot of observations be made from these numbers. For example, with the first two features produced by Ey digits can be classified with a top-1 accuracy of 94%, while with the first two features produced by ED the top-1 accuracy is only 13%, just bit better than random guess.
C ABLATION STUDY
We provide additional details method in Section 3 demonstrate their offectiveness.
Table 7: Top-1 accuracy
Encoder
By writer
By digit
Encoder
By writer
By digit
Ew
0.000454
0.94
Ew
0.001400
0.94
ED
0.005331
0.13
ED
0.015424
0.23
Ey 0.000846
0.70
Ey
0.004946
0.95
(a) First dimensions
(b) First 8 dimensions
17

---

Under review as a conference paper at ICLR 2020
C.1 USING RECONSTRUCTION RESULT AS STAGE CLASSIFIER INPUT
By design, assuming sufficiently powerful networks, the stage 1 method in (Chou et al. 2018 should reconstruct 2 perfectly with C(E(z),S(a)) while at the same time not include any style information in F(x). But in reality, this method worked poorly in our experiments. We discuss our conjecture here.
We think the problem is that the distribution of E(x) is unconstrained and the encoder- decoder can exploit this to trick the classifier while still encode style information in E(x). Assune that the last layer weight of E()) is W1 E Rmxd and the first layer weight of G(:.-) (connected to its content input) is W2 E Rdxx where d is the length of the content code and n and m are the number of features in relevant layers. Then for any invertible matrix H E Raxd replacing W1 with WiH and W2 with H H would give us an autoencoder that computes the exact same function as before but with a linearly transformed distribution in the code space. Note that such a "coordination" is not limited to be between the innermost layers, and the parameters of E(-) and G(+,) may be altered to keep the reconstruction almost unchanged but at the same time transform the code distribution in complicated ways beyond simple linear transformation.
As a result, the encoer-decooee can transform the code distribution constantly during training. The classifier would see different input distribution in each iteration and thus fail to learn to infer the artist from style information that could possibly be included in the output of E(:); and thus would be ineffective at preventing El.) from encoding style information.
Note that constraining the code distribution of the whole training set does not help. For example, if we constraint the code distribution to be standard normal as in VAE, we can take H in the previous example to be an orthonormal matrix. The transformed distribution on the whole dataset would still be standard normal but the distribution of artworks by each different artist would change.
We noticed that this problem would be alleviated if instead of the code E(x), C1(+) tries to classify the reconstruction result ((E()), S(a)) since the output distribution of the docoder is "anchored" to the distribution of training images and cannot be freely transformed. But the output of G(--)), being an reconstruction of the input image, will inevitably contain some style information, conflicting goals between reconstruction and not encoding style information. The important observation is that, the input given to C1()) is allowed to contain style information as long as it is not the style of the real author of x! Thus we arrived at our proposed method where the classifier sees G(E(x), S(a'), the image generated by the encoder's output on x and the style of an artist a' different than the author of x.
giving G G(((z), S(a)) as input to Ci(+) will again cause the problem of
Strictly speaking, by classifying the output of the generator we can only ensure that E(-) and G(s-)) would jointly not retain the style of x in ((()))))(())) E(-) may still encode style information which is subsequently ignored by G( To solve this, we constrain the output distribution of E(.) with a KL-divergence loss as in VAE. With such a constraint,
E(-) would be discouraged from encoding information not used by G(.,) since that would not improve the reconstruction but only increase the KL-divergence penalty.
To compare the effect of classifiers seeing different inputs, we train an alternative version of stage 1 using the method in (Chou et al.) 2018). where instead of a convolutional network that gets the generator's output, we use a multi-layer perceptron that gets the encoder's oulput as the classifier. We use an MLP with 10 hidden layers, each having 2048 features. We then compare G(F(x),, 0) with G(E(s), S(a)), that is, image generated with the content code of an image I' plus an all-zero style code versus the correct style code. A successful stage 1 encoder-decode should reconstruct the input image faithfully with the correct style code but generate style-nutral images with the zcro style codc. We also give the roconstruction of an conventional VAE for reference. The result is shown in figure 10
As autoencoders are in general bad at capturing details, the readers may want to pay attention to more salient features like size of the eyes, shape of facial contour and amount of blush. We can see that these traits are clearly preserved in the style-neutral reconstruction
18

---

Under review as a conference paper at ICLR 2020
Input sample
VAE
Classifier sees B(a))
Classifier sees ((d)) ([E(s), (())
8(()) ((E()),
0)
Figure 10: Comparison of stage image reconstruction with correct style and zero style, using different methods. Column 1: images from the dat aset. Column 2: VAE reconstruction. Column 3: ML. classifier, correct style. Column 4: MLP classifier, zero style. Column Our classifier, correct style. Column 6: Our classifier, zero style. Training images courtesy of respective artists. From top: Azumi Kazuki, Namori, lizuki Tasuku, Tomose Shunsaku.
when an MLP is used for classifying E(x). Our method, while not able to completely deprive the output of style information from the imput in the stylenneutaal reconstruction and also altered the content by a bit, performed much better, and did so without compromising the reconstruction with the correct style.
We point out that this is not due to the MLP not having enough capacity to classify E(x): when the encoder is trained to only optimize for reconstruction and ignore the classifier by setting λo; = 0, the classifier is able to classify E(x) with higher than 99% lop~1 accuracy.
As a direct evidence in support of our conjecture that the performance of stage 1 with ML.P classifier was hurt due to instabilities of the encoder's output distribution, we look at the network trained on NIST with writer label in section B.2 again and track the change of the encoder's output distribution. For comparison, we repeat the training but substitute the classifier for a 6-layer MLP. As reference, we also track the output distribution of a vanilla VAE. The result is shown in figure [1] The encoders' distribution were visualized from iteration 200,000 to iteration 300,000 al. an interval of 20,000 iterations.
The encoder's distribution when an MLP classifier is used was clearly unstable. This is not an intrinsic property of autoencoders, as the output distribution of the vanilla VAE encoder remained essentially unchanged, so the instability must have been introduced by the adversarial training against the classifier. In contrast, the output distribution of our stage I encoder was much more stable. Although the distribution did change, each digit class only fluctuated within a small range and the relative position of the classes remained consistent.
We can also see that although the encoder trained with MLP classifter is still better at clustering the same digit than VAE, it did not separate between different digits very clearly.
C.2 ADVERSARIAL STAGE 2 CLASSIFIER
One important difference in our style-conditional GAN training compared to previous class- conditional GAN approaches is that, in our case, not only is the discriminator adversarial
19

---

Under review as a conference paper at ICLR 2020
(a) VAE encoder
(b) Stage 1 encoder with MLP classifier
(c) Stage 1 encoder with our stage 1 classifier
Figure 11: Change of output distribution of different encoders
but the classifier well. The classifier explicitly tries to classify samples generated using an artist a as "not by a".
The rationale is that, by just correctly classifying the real samples, the classifier may not be able to learn all the aspects about an artist's style. Imagine the hypothetical scenario where each artist draws the hair as well the eyes differently. During training, the classifier arrives at state where it can classify the real samples perfectly by just considering the hair and disregarding the eyes, There will then be no incentive for the classificr learn about the style of the eyes since that will not improve its performance.
If the classifier is adversarial, this will not happen: as long as the generator has not learned to generate the eyes in the correct style, the classifier can improve by learning to distinguish the generated samples from real ones by the style of their eyes.
In general, non-adversarial classifier only needs to learn much necessary to tell different artists apart, while an adversarial classifier must understand an artist's style comprehensively.
To study the effect of this proposed change, we repeat stage 2 but without adding LCg-take to the classifier's loss.
In figure 12 we generate samples using this generator trained without adversarial classifier, from the exact same content code and artists as in figure 2 While we leave the judgment to the readers, we do feel that making the classifier adversarial does improve the likeness.
The choice of "negative log-unikeeihoodd" as the classifier's loss on generated samples might seem bit unconventional:
v.) NLU(y, (NUU(C(CGEEE))S()))....)
LCs-fake
E xmp(a)) a'mp(a)
To many, maximizing the negative log- bieeliil would be a much more natural choice. The major concern here is that the negative likeliioooood is not upper-bounded. Suppose that C2(-) is trained using such an objective. There would then be little to be gained by trying to classify real samples correctly because LC is lower-bounded. To decrease its
20

---

Under review as a conference paper at ICLR 2020
Figure 12: Images generated from fixed style and different contents, when stage 2 classifier is not adversarial.
loss, it would be much more effective to increase the negative log- likelihood generated samples. The result is that the C2(-) simply ignores the loss on real samples while the negative log-likelihood
generated samples quickly explodes. If classifier cannot classify the real samples correctly, one would not expect the generator to learn the correct styles by training it against that classifier.
In contrast, in stage 1 when we trained the encoder E(-) against the classifier Ci(), we maximize the negative log-likelihood. The same problem does not occur here because it is E()) and G(;,)), not C1(+), who tries to maximize it. Considering that the pixel values of the generator's output has a finite range, for fixed classifier, the logelikeli bounded.
Indeed, in stage the negative log-un onguunlikelhoodd would not be a favourable choice: in the equilibrium, the classifier is expected to perform poorly (no better than random guess), at which point the negative log-unliklihhod would have tiny gradients while the negative log-likel would have large gradients.
A Note on Using Classification Accuracy as Quality Mcasure. From the discussion above, we conclude that the accuracy of classification by style the generated samples with an independently trained classifier cannot properly assess the level of success of style transfer: since such a classifier, trained solely real samples, cannot capture complete style information, a high accuracy achieved by generated samples does not indicate that the style transfer is successful, It likely, though, that the generator of bad quality if the accuracy is very low.
However, we feel that if the generator could score a high accuracy even against an adversarial classifier, it must be considered success. Unfortunately, our method has yet to achieve this goal.
In table S we report the classification accuracy by both the adversarial and the non-adversarial classifier on samples generated both by the generator trained against the adversarial classifier and by the generator trained against the non-adversarial classifier. Given that for the majority artists the number of artworks available is quite limited, to use the data at maximum efficiency we used all data for training and did not have separate test set. As substitute, we generate samples from content codos drawn randomly from the learned content distribution which is modeled by multivariate normal distribution over the whole training set. Notice that during stage training we only use E(x) where z is from the training set and F(-) is fixed, the work only ever sees fixed finite set of different content codes.
21

---

Under review as a conference paper at ICLR 2020
Table 8: Top-1 classification accuracy of two classifiers on samples generated from two generators
Adversarial C2
Non-adversarial C2
G trained with adversarial C2
14.37%
86.65%
G Irained with non-ad versarial C2
1.85%
88.59%
Figure 13: Images generated from fixed style and different contents, when explicit condition on content is removed.
Thus, random drawn content codes are indeed unseen which serves the same purpose as separate lest set.
In each test, the same number of samples as in the training set (106,814) are generated such that the content codes are independent and random and each artist appear exactly as many times in the training set..
As can be seen, there is huge discrepancy between the classification accuracy scored by the different classifiers. If trained to do so, the adversarial classifier can easily tell that the generated samples are not of the correct style, so extreme caution must be exercised any conclusions is to be drawn from the classification accuracy by non-adversarial classifier.
C.3 EXPLICIT CONDITIONING ON CONTENT IN STAGE 2 GENERATOR
Our treatment content in stage is also quite different, in that we use a content loss to explicitly condition the image generation content. Compare this to (Chou et al. 2018 where there no explicit condition on content, rather, the stage generator's output is added to the stage decoder that the stage condition content, which is guaranteed by minimizing the reconstruction loss, is inherited by stage 2.
While did not experiment with their approach, show here that in framework the explicit condition on content by content loss necessary. We repeat stage 2 training with Acont 0 that (x)(E(x). S(a')) is not constrained to be close E(x). We then examine if style and content can still be controlled independently. Images generated by fixing the style and varying the content is shown figure 13 while images generated by fixing the content and varying the style is shown in figure
We can clearly see that the content code has lost most of its ability to actually control the content of the generated image, while the style code would now control both style and content. Curiously, the content code still scems to control the head pose of the generated character, this is the most noticeable variation in figure 13 while in figure 14 the characters had essentially the same head pose.
Figure 14: Images generated from fixed content and different styles, when explicit condition content is removed.
22

---

Under review as a conference paper at ICLR 2020
This can be considered form of "partial mode collapse": while the network is still able to generate samples with rich variation, some of the input variables have little effect on the generated image, and part of the imput information is lost. This happens despite the fact that the stage generator is initialized with the weights of the final stage 1 generator, in which the content code is able to control the content. So, this is not the case where the number of input variables is more than the dimension of the distribution of the dataset so that some imput variables never got used. Rather, the ability control content initially present but subsequently lost. So, in our approach, an explicit condition on the content by content loss is necessary.
23


Reviews:
[{"review_id": "rklNwrXSqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a Generative Adversarial Disentangling Network based on two stages training the first aims at learning a style independent content encoder and then content and style conditional GANs is used for synthesis.\nAt stage 1 training to prevent the encoder from encoding authors introduce a gan style training in which an adversarial classifier that tries to predict the corresponding artist from the encoded image.\nAt stage 2 is training a style/content conditional gan. To condition on the style (artist) authors introduce an extra adversarial classifier so the generator tries to generate samples that would be classified as the artist that it is conditioned\non. While to condition on the input content another loss is ensuring that the generated image is encoded back to its content input.\n\nAuthors compare the proposed method against the original neural style transfer and StarGAN over various styles within the context of anime illustrations and the NIST Dataset where styles are being represented by artist name. \n\nWhile the work tackles some of the problems by conditioning only on artist names other than style features that might be hard to have annotations for. The proposed modifications are quite incremental. Additionally, the experiments section is quite weak, evaluation is only done quantitatively over some cherry-picked examples, although some extra ablation study in the appendix is provided. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1g81xakcH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n1. This paper proposes a method to learn disentangled style (artist) and content representations.\n2. By carefully designing two-stage training objectives, the method learns a style-independent content-encoding E at the first stage and the style encoder S and generator G both from the first and the second stage.\n3. Empirical results justify the validity of the method.\n\nI think this paper makes a good contribution to disentangle style and content in anime. My main concern is the complicated learning procedure design may affect the reproducibility of this method. Moreover, I will suggest several points to the authors to clarify in the main text.\n\n1. I encourage the authors to release their code when published.\n\n2. In stage 1 (Style Independent Content-Encoding), the purpose of the classifier C, to my understanding, is to try to classify the generated example G(E(x), S(a')) as the \"ground-truth\" style (a). That is, the classifier C tries to disregard the S(a') when making a decision. As an adversarial player, E, G, S will try to fool C by making E(x) to be non-informative regarding the style. However, since you are still optimizing G and S, how do you make sure that it is safe to hold E fixed while still changing G and S in the second stage? Or more specifically, how do you make sure the style encoding network S preserves a good one in the second stage? Aside from that, are you using the trained G, S from the first stage to initialize G, S in the second stage?\n\n3. There are eight different terms in stage 2, so it worth checking the necessity for those terms. E.g. what happens if you drop the L_cont term? The term L_cont seems to guarantee the validity of E, but E is fixed in step 2.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkeldOzRtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an image generation method with the focus on generating anime faces from various artists. The proposed method which is a combination of conditional GANs and conditional VAEs manages to generate high fidelity anime images with various styles.\n\nThe paper is well-written and easy to follow and understand. The goals are clearly stated and the background (which is more of history), as well as related work, is comprehensive. The decision behind every design decision has been mentioned in detail which makes the paper stronger. The main writing flaw of the paper is in the figures where more annotation and caption is required to make them easy to understand. For example:\n- The significance of colors in Figure 1 (architecture of the model) is not annotated at all and the letters are not clear either (although they are described in the text itself a figure should be comprehensive by itself).\n- Figure 2 and Figure 4 are really hard to understand with very limited annotation and caption. I had to read the text multiple time to Figure out what is what in these figures which is not a good sign for clarity.\n\nIn terms of experiments, I think where the paper suffers the most is in comparison with other conditional methods. In Section 5 it has been clearly mentioned that \"this result can be expected from a class-conditional GAN and the focus in on Disentanglement\" however very little evidence has been provided for superior disentanglement. More experiments are required to demonstrate the capabilities of the model compared to other conditional methods (which is currently only limited to StarGAN) as well as its capability of disentanglement. I agree with the authors that quantitative evaluation of generated anime faces is not easy (although it is possible with a carefully designed human study), however, the disentanglement (which is the focus of the paper) is easy to evaluate quantitatively. This demands for more experiments on disentanglement datasets with known generative factors. Although the current ablation study in the Appendix provides more details for architectural decisions, a more qualitative and quantitative comprehensive ablation study (by actually ablating the final model) can help to demonstrate these decisions.\n\nIn conclusion, the paper has great results. We all know a big part of writing this kind of paper is to make the model \"work\" and authors truly demonstrate that they worked hard. However, the impact of the paper (in the current form) is not clear. With the focus on disentanglement, little evidence has been provided to justify the capability of the proposed method. I believe by addressing my comments on the experiments the paper can be easily pushed above the acceptance bar. Also releasing the code dataset should increase the impact of the paper.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a two-stage adversarial training approach for learning a disentangled representation of style and content of anime images. Unlike the previous style transfer work, here style is defined as the identity of a particular anime artist, rather than a set of uninterpretable style features. This allows the trained network to generate new anime images which have a particular content and are drawn in the style of a particular artist. While the approach works well, the reviewers voiced concerns about the method (overly complicated and somewhat incremental) and the quality of the experimental section (lack of good baselines and quantitative comparisons at least in terms of the disentanglement quality). It was also mentioned that releasing the code and the dataset would strengthen the appeal of the paper. While the authors have addressed some of the reviewers’ concerns, unfortunately it was not enough to persuade the reviewers to change their marks. Hence, I have to recommend a rejection.
Decision: Reject