Copy the Old or Paint Anew? An Adversarial Frameworkfor (non-) Parametric Image Stylization

Nikolay Jetchev

Urs Bergmann

Gokhan Yildirim

Zalando Research
Berlin, Germany
nikolay.jetchev@zalando.deZalando Research
Berlin, Germany
urs.bergmann@zalando.deZalando Research
Berlin, Germany
gokhan.yildirim@zalando.de

Abstract
Parametric generative deep models are state-of-the-art for photo and non-photo realistic
image stylization. However, learning complicated image representations requires computeintense models parametrized by a huge number of weights, which in turn requires large
datasets to make learning successful. Non-parametric exemplar-based generation is atechnique that works well to reproduce style from small datasets, but is also computeintensive. These aspects are a drawback for the practice of digital AI artists: typicallyone wants to use a small set of stylization images, and needs a fast flexible model inorder to experiment with it. With this motivation, our work has these contributions: (i) anovel stylization method called Fully Adversarial Mosaics (FAMOS) that combines thestrengths of both parametric and non-parametric approaches; (ii) multiple ablations andimage examples that analyze the method and show its capabilities; (iii) source code thatwill empower artists and machine learning researchers to use and modify FAMOS.
Tiling of small stones was a classical ancient art form, and in modern times there are efficient algorithms
to produce such mosaics (with non-overlapping tiles) digitally [10]. Seamless mosaics in the style of theRenaissance painter Archimboldo are more challenging, but modern deep learning methods allow efficientseamless image stylization. Neural style transfer [4] uses filter statistics (pretrained on a huge dataset) ofa style image to optimize an output image. [8] learns a texture process adversarially from multiple input
images and then optimizes samples from it to paint a mosaic. [7] can stylize images in one forward pass ofthe generator network, but requires for training paired images from 2 domains. [17] is more flexible and cantranslate unpaired images. However, the authors note that only local stylization of color and texture is learned,
larger geometric changes are not easily achieved. In general, parametric approaches to image generation havecapacity issues when very complicated image styles are considered.
Non-parametric image quilting [3] combines patches from texture data in order to smoothly reconstruct atarget image – "texture transfer". The work of [13] uses patch similarity and content copying in order to solve
an image analogy problem. The work of [12] uses neural patch matching (hard attention) combined with acontent loss. Their results are visually impressive, but expensive, and for one output image only. Both [12, 13]rely on pretrained networks and this can negatively impact the image quality if the texture and content image
distributions differ too much. As a further drawback, all of [3, 12, 13] are slow due to nearest neighbours
routines used for patch search, and cannot scale to large images.
There are already methods to improve conditional parametric image generation with a non-parametric memory[1, 16], but they are more specific for datasets of paired images, and are not directly suitable for artistic
applications of image stylization. These methods are also slow to optimize a single output image – thebottleneck is the need to perform expensive search in the database.
Our method, called FAMOS, combines the advantages of parametric methods (fast to infer stylization once
trained on data) with those of non-parametric methods (accurate reconstruction of complex styles) in a GAN[5] framework. We learn a fully convolutional network that can predict stylization images of very large size.
In Figure 1 we show the architecture of the generator G, which is the key novelty. A U-net [7] is used to
predict a mixture matrix A used to create image IM copied from input style data, a mask α and an image
IG . They are used to blend and fine tune the final output I = α IG + (1 − α) IM . This structure allows

32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.

True/fake ?

True/fake ?

Generator:
Template mix+blending

Discriminator:
patch discrimination

Figure 1: The architecture of the generator G: a single Unet predicting the template mixture weights A, an image IG
and blending mask α. The discriminator predicts true/fake looking texture patches.

Figure 2: FAMOS outputs a stylization of a Rembrandt portrait as content image, 2000x2400 pixels. Style images: 8
Google maps views of Amsterdam, and 7 paintings of C. Attersee (4 shown). We can emphasize more or less the texture
aspect or content reproduction by tweaking the loss hyperparameter (λ = 3 and λ = 1 shown for Amsterdam).

our network to adaptively fill some regions of the final image with generated content similar to a traditional
methods (e.g. [7]) and for other regions decide to directly copy from the source style data.
For training FAMOS we have as input for the discriminator: (i) a set of content images IC of various sizes,
from which we can define the cropped patch distribution Ic ∼ PC of fixed size H × W ; (ii) style (texture)
images IT with crop distribution It ∼ PT . For the generator, we crop from the memory M (see below)
patches (M 0 , ψ) ∼ PM . Here we keep also the crop grid ψ, of size 2 × H × W . For training FAMOS,
we will use minibatches of patches, and for inference we can roll-out on any image size (due to the fully
convolutional generator). The loss function combines adversarial and content reconstruction terms:
Ladv (G, D) = EIt ∼PT log D(x) + EIc ∼PC ,(M 0 ,ψ)∼PM log(1 − D(G(Ic , M 0 , ψ))),(1)
Lcontent = EIc ∼PC ,M 0 ∼PM kφ(Ic ) − φ(G(Ic , M 0 ))k2 ,G∗ = arg min arg max Ladv + λLcontent .
G

D

(2)
(3)

The correspondence map φ in Lcontent specifies the perceptual space for the reconstruction term [3, 8].
The use of a memory M ∈ RN ×3×H×W is a novel non-parametric aspect of our model. It is created byrandomly sampling N images from IT and interpolating each of them into a tiling of a fixed size. Eachtiling has a random initial grid coordinate offset, and we may also call them templates. The mixture matrix
A ∈ RN ×H×W is calculated as a function of the content patch Ic and the memory template coordinates ψ. In
a sense, ψ encodes the available template patches (given the offset) with which we can paint at that spatial
position of the convolutional input. Since we use a set of tiled templates M , this gives enough information to
the network to predict a good mixture of coefficients to reconstruct the content and fool the discriminator.
We apply softmax to A (in the dimension of the N templates) to get Ã, and use it to create the soft attention
memory image IM via batched dot operation Ã ⊗ M = IM ∈ R3×H×W . While somewhat analogous tovarious attention-like methods [15], it is applied only to fixed spatial positions – e.g. spatial position h, w
in the output IM results from copying values from the same position h, w from M . This aligned spatial
structure is a cost efficient alternative to full attention or to nearest neighbour neural patch best match search,
O(HW N ) vs O(H 2 W 2 ). Note that each tiling is translated by a random offset, so if N → ∞ our module
will be expressive as full spatial attention. However, in practice a small N is enough to construct an expressive
yet not too memory hungry memory module.
In summary, FAMOS is an image stylization approach with a new concept, with the following key properties:(i) generation of seamless mosaics with unique visual aesthetic, a single neural model that can decide whether
to copy or generate parametrically different regions of the output image. (ii) a computationally efficient
non-parametric memory module that allows to copy complex textures that are difficult to represent in aparametric model; (iii) the fully convolutional model can create very large images at inference time – allcalls to G can be efficiently split into small chunks seamlessly forming a whole image, without memory
constraints [8, 9]; (iv) a flexible framework that can fit many artistic choices given style and content input
data. The result of training FAMOS and inferring a final mosaic on a large content image is shown in Figure 2– zooming-in reveals how well style image details are preserved. See the Appendices for more generative art.
2

References[1] Aayush Bansal, Yaser Sheikh, and Deva Ramanan. Pixelnn: Example-based image synthesis. CoRR,
abs/1708.05349, 2017.
[2] Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. Learning texture manifolds with the periodic
spatial GAN. In Proceedings of The 34th International Conference on Machine Learning, 2017.
[3] Alexei A. Efros and William T. Freeman. Image quilting for texture synthesis and transfer. In Proceedingsof the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, 2001.
[4] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style. CoRR,
abs/1508.06576, 2015.
[5] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair,
Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information
Processing Systems 27, 2014.
[6] Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved
training of wasserstein gans. CoRR, 2017.
[7] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation withconditional adversarial networks. CoRR, abs/1611.07004, 2016.
[8] Nikolay Jetchev, Urs Bergmann, and Calvin Seward. Ganosaic: Mosaic creation with generative texture
manifolds. CoRR, abs/1712.00269, 2017.
[9] Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative
adversarial networks. CoRR, abs/1611.08207, 2016.
[10] J. Kim and F. Pellacini. Jigsaw image mosaics. In Proc. of the 29th Annual Conference on Computer
Graphics and Interactive Techniques, SIGGRAPH, 2002.
[11] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,
2014.
[12] Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks forimage synthesis. In CVPR, pages 2479–2486. IEEE Computer Society, 2016.
[13] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual attribute transfer through deep
image analogy. ACM Trans. Graph., 36(4):120:1–120:15, 2017.
[14] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional
generative adversarial networks. CoRR, abs/1511.06434, 2015.
[15] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CVPR,
2018.
[16] Jiaya Jia Xiaojuan Qi, Qifeng Chen and Vladlen Koltun. Semi-parametric image synthesis. 2018.
[17] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation
using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International
Conference on, 2017.
[18] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli
Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information
Processing Systems 30, 2017.
3

Figure 3: A content image of certain two famous people stylized with an aerial view of downtown Los
Angeles. Mosaic image size 1920x1080 pixels. Image created with a FAMOS model without memory
templates – the regular nature of the city grid allowed nice results with the convolutional Unet generator.

Appendix I: Exploration of FAMOS Possibilities
Tweaking FAMOS: parametric vs non-parametric modules Our tool can deliver good looking mosaics,
but it is still a tool dependent on experimentation done by the artist. The right choice of texture and content
images is a part of the creative search for a good artwork. The selection of the neural architecture and thehyperparameters can also change the final result a lot. Here we give a small summary of all the choices theuser of FAMOS can do in order to influence the final artwork:• choice of content and texture image sets, potentially scaling them to tweak scale of visual details
relative to generator receptive field (see [9] for examples). We get best results when using few style
images with consistent texture properties (repetition of small details), but large sets of diverse images
may also work.
• the most important choice: keep only the parametric or non-parametric generative modules of the
network, discussed in detail in the current section.
• layer count and kernel size of the G and D networks, changing the receptive field of the network• Correspondence map for image reconstruction distance metric – see below• count of memory templates N and choice of tiling mode (see below)• regularization of mask α, affecting how much the FAMOS relies on copying non-parametrically andparametrically. See Appendix II for more on this and other regularization terms we considered.
• stochastic noise – we add also noise to the bottleneck of the Unet, see [2] for some intuition how
this can change the generated output.
We share the links for two online galleries we prepared with many additional examples of the mosaics thatcan be created with the FAMOS model:• Gallery 1: pure convolutional generation (memory module disabled). When the style images havetexture-like properties, this architecture is also an interesting tool and can create good looking output
image stylizations (texture mosaics), while being faster computationally. Please see Figure 3 for an
example of the capabilities of that mode of FAMOS.
• Gallery 2: non-parametric mosaics with FAMOS which make use of the memory module – thisallows to copy flexibly parts of the style images when necessary.
Figure 5 shows the drastic differences in the output of the generative model from Figure 5 if we emphasize
only some parts of it. If we disable the memory M and have N = 0 templates (equivalently if α = 1 in theblending equation), we end up with a model very close to a traditional Unet generator, purely parametric
generation of IG . While in some cases this can be quite efficient as well, especially for easy to learn repetitive
textures, it can fail for more complicated image styles such as the Santorini island one.
4

Figure 4: The image of the island of Santorini from commons.wikimedia.org we used as a complex
stylization (unconventional texture since scale varies widly). The content image of a fashion model, fromwww.zalando.de.

Figure 5: Different outputs from different variations of the architecture. From left to right: (i) only parametricimage generation, (ii) nonparametric mix without blending, (iii) blended parametric and non-parametric as in
the full FAMOS architecture. Images of size 1536x1024 pixels, best seen zoomed-in.

We can also force α = 1 and keep entirely the template image IM , a non-parametric behaviour. This canpreserve well the details of Santorini, but has visual glitches: some hard edges, blurriness at borders ofdifferent template mixing regions.
The full output of FAMOS, the blended refined output has the best image quality in our opinion, keeping thedetails of the non-parametric mix and correcting some of the artifacts there.
However, as might be expected of an artistic digital algorithm with so many options, depending on thecomplexity of the stylization image distribution, any of the 3 variants can be an effective tool of artistic
expression.
Tiled memory templates The memory templates M are created by moving a regular coordinate grid
Θ = [−1, 1] × [−1, 1] from style image IT . We can use either mirror or wrapping padding mode wheninterpolating using a coordinate grid η + Θ ∈ R2×H×W , where η ∈ R2 is a random offset added to Θ.
In mirror mode (a.k.a. reflect mode), coordinate values 1 +  are smoothly decreasing again and theinterpolation routine would copy pixels from positions 1 − . In mirror mode we are simplifying the task ofthe prediction module, since the templates tile neatly into each other without hard borders. This also has avery interesting visually kaleidoscope-like aesthetics because of the axes of reflection in the appearance of thetiled templates.
In wrap mode, 1 +  coordinates are hard reset and copy pixels from −1 + . In wrap mode, the model should
learn and adapt to avoid borders, which will be penalized by the discriminator.
5

Figure 6: Using the Santorini panorama image as style (size 1270x780 pixels ) tilings of size 1024x1536
pixels (size of the given content image IC ) are interpolated, 2 images for wrap and mirror mode illustration.
Figure 6 illustrates this process, showing wrapped and mirrored mode, and 2 interpolations with random
coordinate offsets to illustrate how this translation enriches the memory tensor M and allows our fixed spatial
position memory module to copy varied content.
Correspondence map The choice of correspondence map used for reconstruction also matters, as discussed
by [4] who selects different layers of the pretrained VGG-19 network, or [8] where the effects of dowmsamplng
are discussed. For FAMOS it worked well with any of these settings:• convolve with a Gaussian filter (using reflection padding to avoid border artifacts) and convert togreyscale• downsample image 4,8 or 16 times and convert to greyscale• train a small reconstruction conv. network with kernels 1x1 and stride 1 so that the loss becomeskIC − φrec (I)k – effectively this converts from the color space of the textures to the content image.

Appendix II: Training and Implementation Details
Our code would be released in GitHub at https://github.com/zalandoresearch/famos after publication at a conference or workshop.
Network and training We used typically N = 100 memory templates for M . Note that these can take a
lot of memory if we have large spatial size H, W , but they can stay in RAM – only patches from them areshifted to the more limited GPU memory. Both generator and discriminator use batch norm and kernels of
size 5x5. We use ReLU and leaky ReLU nonlinearities. To avoid checkerboard artefacts we use upsamplingconvolution, instead of transpose convolution. The training patch size was 160 – but deeper Unets would
required larger patch size; the minibatch size was 8. The typical channel (width )and layer (depth) counts weused for generator and discriminator are shown in Figure 7. Standards architectures as described in [7] canalso work. For training we use DCGAN loss [14] and the ADAM [11] optimizer.
Our code is implemented in Pytorch, and we ran it on a single NVIDIA p100 gpu card. The time to get first
nice image is a few minutes usually, but several hours can be required to train fully a complicated model.
Note of caution: training instabilities We also note in general that the FAMOS architecture can sometimes
diverge or require some tuning given different content and style images. The mixing properties of the texture
process of the style image are very important (see for some discussion [2]). Such texture properties determine
how easy it is to sample and generate – a simple texture (e.g. rice particles) is much easier to use than anon-texture image such as the island of Santorini from Figure 4. The interplay of generation and curation on
the human artist side is an essential part of generative art, as a human counterpart to the adversarial nature of
the GAN game between generator and discriminator. Changing the parameters or restarting can often lead
to better results. As with many GAN models, early stopping may be useful: the user of FAMOS can saveregularly the output of the model and keep those images that seem most promising.
6

Decoder(N+4)x160x160
128x80x80
256x40x40
512x20x20
512x10x10

Discriminator

512x10x10

Encoder

256x20x20
128x40x40
64x80x80

256x20x20
128x40x40
64x80x80
3x160x160

5x160x160

Figure 7: Channel and layer counts for our standard FAMOS architecture. We use typically 4 layer deep
encoder and decoder (in Unet generator), and 4 layer deep discriminator. The generator takes content image
IC and coordinates ψ as input, and outputs the mixture array A, the blending mask α and image IG to use forblending. The total number of output channels is N + 4, given N memory templates.
A particular failure mode can happen if the mixing matrix A collapses: few templates from M are always
used to generate IM and the other templates are ignored (i.e. entropy of A close to 0). In that case, theparametric part of FAMOS can still paint a nice image on top of that IM serving as background canvass, but itwould be preferable if FAMOS can use the full expressive power of it memory templates. The other extreme
case happens rarely: the entropy of A stays high and all templates mix to a greyish image IM . This may
require special tuning of an entropy regularization schedule.
We acknowledge these convergence issues, but even so we think that FAMOS is an interesting novel image
generation method, and is a fun tool to use and explore. Some regularization terms can help stabilize trainingof FAMOS, but the exact research of the "right" regularization terms and weight in the loss is left to future
work. We considered these terms as part of a regularization loss Lregularize for the generator.
• small entropy of mixture matrix A – this will force values to be 0 or 1, allowing to cleanly select andkeep some memory template, avoid blurriness• small total variation of A – smooth changes• small norm of blending mask α – i.e. forcing that we are close to IM , the true memory templates,and paint less with the parametric GAN. Thus will force to only paint details with the GAN Unet
when copying from memory does not work• small total variation of α
While it is not entirely clear when these terms stabilize training, they are interesting on their own as additional
controls from side of the artist user of FAMOS. The next paragraph has more comments on the stability andperformance of the method.
Practical tips: what works and does not work In order to find a good architecture for FAMOS, we tested
a lot of architectural choices. Some work better than others.
• If the cropping coordinates ψ are identical for content and template image patches distributions
PC , PM – generalization to out-of-sample content images may deteriorate since the network learns
by heart that some content patches go together with some template patches. To remedy this, cropping
content patches and memory templates at different random locations allows better generalization.
• Downsampling and copying the coordinates ψ after every pair of conv+batch-norm layers makes
FAMOS better when using this croppng mode for generalization. This effect is subtle and needs
more investigation, but we assume that it makes the network more sensitive to the cropped template
offset location.
• However, if we want to train really well just mosaics for the training content image set, and do
not need to generalize to additional out-of-sample content images, then we can crop the "same"
coordinates from M and IC , both for training and inference, thus allowing the network to learnmuch better the spatial relation between content and memory template, and give better mosaics
result. In a sense, such a mode is analogous to optimization based stylization [4, 8], since only the
result on a single image matters. However, even in that case FAMOS remains a very performant
model capable of dealing with very high resolution content images.
• We use a single Unet to predict the N channels of A for mixing templates and the 4 channels ofα, IG for blending – this additional weight sharing works and is much faster than having a design
7

True/fake ?

True/fake ?

Generator

Discriminator:
patch discrimination

Figure 8: An alternative architecture of the generator G with a chain of non-parametric and parametricmodules. The discriminator of true/fake looking texture patches is standard patch-based one [7].
with 2 Unets as in Figure 8. But further experiments may find cases when more capacity (e.g. byadding residual layers) can improve image quality, as is often the case for GAN methods.
• WGAN-GP [6] loss gives better convergence to FAMOS than DCGAN loss [14] when having
nonparametric memory M . However, if we disable the memory and rely only on the parametric part(e.g as in Figure 3) than DCGAN is much better. This inconsisteny which loss is best was quite
surprising for us, but we did not investigate it in detail.
• Adding some noise channels (identical spatially and of spatial dimension as IC ) to the input of
the Unet (concatenated to IC , ψ) or to the bottleneck, seems to stabilize the behaviour of the Unet.
However, the outputs are not very sensitive to the stochastic noise inputs – some techniques [18]exist to ensure that conditional GAN methods have multimodal outputs.
• If M is an input to the generator as well, 3N more channels when having N templates – no gain in
quality, only downside of more computation required.
• We use randomly translated copies of the textures in M and duplicate the images with different
offset. An alternative can be to predict the morphing to each of the texture images and avoid having
random duplicates. We tried directly morphing with optical flow – there were issues with poor
gradients of the loss and deformations of the style images, so we did not pursue that option.
• If use only a supervised content loss without GAN loss for the memory mixing module – poor
results, the GAN loss is required indeed.

Appendix III: Outlook
As stated in the previous section, we are examining in detail various model convergence issues and testing
regularization terms that can improve stability of FAMOS.
In the future, we plan to investigate the ability of our model to generalize to new content and memory images
at inference time. The capacity of our model (channels of the Unet) with respect to the template memory size
N can be also examined.
We currently mix the memory templates only at a single scale using RGB pixels. Previous work [12] has used
multiple scales and feature spaces other than RGB for copying – this can be added as improvement in the
FAMOS framework.
The U-net we use for mixing coefficients prediction can be replaced with an attention-like structure [15],which can further improve the generalization of the FAMOS model, allowing to use at inference time many
additional texture templates. Our first results seem promising in that direction.

8