Evolution of Style Transfer techniques
Tags: Neural Style Transfer, Computer Vision, Artificial Intelligence, Cyclegan, Deep Learning

What is style transfer?

Neural Style Transfer refers to a class of software algorithms that manipulate digital images, or videos, in order to adapt the appearance or visual style of another image. Neural Style Transfer algorithms are characterized by their use of deep neural networks for the sake of image transformation. Simply put output image retains the core elements of the content image but appears to be painted in the style of the style reference image.

The Deep Learning magic

Our starting point was the original algorithm for neural style transfer by Gatys et al (2015) based on optimizing an image to match the content and style of another image by manipulating some clever losses.

In summary, the general idea is:

Take an image for style ( style_img )

) Take an image for content ( content_img )

) Take a pre-trained neural net

Create an “output” image where each pixel is a parameter to optimize

Optimize the “output” image so that the content loss and style loss are minimized

Networks

Convolutional neural network

In deep learning, a convolutional neural network is a class of deep neural networks, most commonly applied to analyzing visual imagery. They are also known as shift invariant or space invariant artificial neural networks, based on their shared-weights architecture and translation invariance characteristics. Convolution neural networks work well in image feature representation tasks, where content and style features can be separated in the same network because lower-level layers are more concerned with pixel values and image textures and higher-level layer features are more abstract. So we will be using the VGG network for feature representation purposes. Where lower-level layer features are used as a style feature representation and higher-level layer features as a content feature representation.

VGG 16 Network

The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual computer vision competition. Each year, teams compete on two tasks. The first is to detect objects within an image coming 200 classes, which is called object localization. The second is to classify images, each labeled with one of 1000 categories, which is called image classification. VGG 16 was proposed by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group Lab of Oxford University in 2014 in the paper “VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION”. This model won 1st and 2nd place in the above categories in the 2014 ILSVRC challenge.

VGG-16

VGG 19 Network

VGG-19 is a convolutional neural network that is 19 layers deep. You can load a pre-trained version of the network trained on more than a million images from the ImageNet database. The pre-trained network can classify images into 1000 object categories, such as a keyboard, mouse, pencil, and many animals. As a result, the network has learned rich feature representations for a wide range of images. The network has an image input size of 224-by-224.

VGG-19

Cyclic GAN

The CycleGAN is a technique that involves the automatic training of image-to-image translation models without paired examples. The models are trained in an unsupervised manner using a collection of images from the source and target domain that do not need to be related in any way.

Cycle GAN

Losses

Content Loss

The content loss is calculated using the target and content images. First, we select one of the layers of our model. Then we feedforward the content image through the pre-trained model and get the feature maps produced by the selected layer. After that, we feedforward the target image and get the feature maps produced by the selected layer.

Then we calculate the loss using these feature maps (content feature maps and target feature maps). For example, we can use the mean squared distance between the content feature maps and the target feature maps as a loss function.

Style Loss

In order to calculate the style loss, we need to define a style representation for the images. In the paper, the style representation is defined as the Gram matrix of the feature maps. The Gram matrix is a measure of a statistic in the features. It measures the correlation between the feature maps.

The style loss is calculated using the style representations (Gram matrix) for the style and target images. In the paper, the mean squared distance is used as a style loss. So the style loss will be the mean squared distance between the two Gram matrices.

Total Loss = content weight * content loss + style weight * style loss

By varying the values of content weight and style weight, we can control the amount of information retained from the content image and style image.

Image style transfer

To apply style transfer on the images, we load the weights of the pre-trained model, extract the content and style features and minimize the loss between the target image and content/style image where the Initial target image is used as a mixup of lower-level layer feature of style image and higher-level layer feature of content image. Before training, we set the weights of layers in decreasing order from lower to higher-level layers to calculate the loss. After training, when the model converges, it produces a style transferred output image.

The above strategy works well when we want to impose the style over the entire content image. To target a specific area in the content image, the Cycle GAN method is used.

Image Style Transfer