Training on Art Composition Attributes to Influence
CycleGAN Art Generation

Holly Grimm
Independent Software Developer and Artist
holly.grimm@gmail.com

Abstract
I consider how to influence CycleGAN, image-to-image translation, by using
additional constraints from a neural network trained on art composition attributes.
I show how I trained the the Art Composition Attributes Network (ACAN) by
incorporating domain knowledge based on the rules of art evaluation and the resultof applying each art composition attribute to apple2orange image translation.

1Introduction

The standard adversarial and cyclical losses of a CycleGAN [1] were augmented with additional loss
terms from a convolutional neural network trained with art composition attributes. During training of
the CycleGAN, the user specifies values for each of the art composition attributes. For instance, if atarget contrast value of 10 is specified, the generator should output images with more contrast than if
the target contrast value is 1.
1.1

Art Composition Attributes

Eight art composition attributes were selected: variety of texture, variety of shape, variety of size,
variety of color, contrast, repetition, primary color, and color harmony. Five hundred images from theWikiArt dataset [2] were labeled with these attributes. Figures 1, 2, 3 and 4 are examples of low andhigh values for variety of texture and contrast.

Figure 1: low texture

2

Figure 2: high texture

Figure 3: low contrast Figure 4: high contrast

ACAN

Training consisted of fine-tuning a ResNet50 [3] pretrained on the ImageNet dataset. ResNet50 is afifty-layer deep residual network with 16 residual blocks. Global Average Pooling (GAP) is applied
to the ReLU output from each of the sixteen ResNet block activations, called rectified convolution
maps [4]. The sixteen GAP outputs were concatenated and L2 normalization was applied to create amerge layer. From the merge layer, there are eight outputs, one for each of the attributes.
32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.

3CycleGAN and ACAN

In addition to the standard CycleGAN losses (Adversarial, Cycle-Consistency, and Identity) theACAN losses are a series of eight losses generated when the translated image is passed through theACAN with eight target values. The difference between these target values and the values output by
the network are the attribute losses.

4Results

Below is a sampling of the results of running the CycleGAN training with ACAN on the apple2orange
dataset. Even with a small training set size of 500 images, the ACAN is able to learn and generateapples with the eight art compositional attributes.

Figure 5: original apple Figure 6: color harmony Figure 7: color harmony Figure 8: color harmony
monochromaticimage
analogouscomplementaryFigure 9: low contrast Figure 10: high contrastFigure 11: low color

Figure 12: high color

Figure 13: low texture Figure 14: high texture Figure 15: low repetition Figure 16: high repetit.

The most surprising result of this project is thepainterly effect that theACAN was able to inject
into the CycleGAN generated images as seen inFigures 18 and 19.
Figure 17: original

Figure 18: translated Figure 19: reconstructed2

Acknowledgments
This project was initially developed as part of the 2018 OpenAI Scholars program. I would like tothank my mentor, Christy Dennison from OpenAI, for her helpful comments along with support fromLarissa Schiavo, Joshua Achiam, Jack Clark, and Greg Brockman from OpenAI.

References[1] Zhu, J. & Park, T. & Isola, P. & Efros, A.A. (2017) Unpaired Image-to-Image Translation using CycleConsistent Adversarial Networks. arXiv:1703.10593.
[2] Nichol, K. (2016) Kaggle dataset: Painter by numbers.
https://www.kaggle.com/c/painter-by-numbers.
[3] He, K. & Zhang, X. & Ren, S. & Sun, J. (2015) Deep Residual Learning for Image Recognition
arXiv:1512.03385.
[4] Malu, G. & Bapi, R.S. & Indurkhya, B. (2017) Learning Photography Aesthetics with Deep CNNs.
arXiv:1707.03981.

3