Making deep learning your artist with Style Transfer
By: Hafeez Jimoh
Tags: Style Transfer, Convolutional Network, Machine Learning, Computer Vision, Deep Learning

“Style is something each of us already has, all we need to do is find it.” — Diane von Furstenberg

Photo by Matthieu Comoy on Unsplash

In order to classify images with CNN, we need to extract the features first and these features are fed into our classifier. These features are not only useful for classification purposes but also for image reconstruction and are the foundation of Style Transfer and Deep Dream. Computer vision algorithm powered by the advancements in deep convolution neural network has given the power to extract textures, edges, colour and other higher-level features of an image and blend these features into another image. This process is simply what we refer to as style transfer.

Tensorflow page defines Style transfer as: “Neural style transfer is an optimization technique used to take two images — a content image and a style reference image (such as an artwork by a famous painter) — and blend them together so the output image looks like the content image, but “painted” in the style of the style reference image.”

What are the contents and styles in the context of CNN and Images?

If we have several layers of CNN stack together, as we go down deeper in the network, our CNN learns to extract more complex features from the image and discard those ones that are noise that is not useful for classifying our image with the help of pooling layers. Thus the last layer of our image in the CNN network would ideally represent the content of the image where details like pixel colours and textures are removed already and we are left with high-level content that represents the object and how it is arranged in the input image. In other words, feature representation in higher layers of the network is referred to as the content representation

Styles in images represent the textures, the colours, and curvatures in an image. The goal of style transfer is to isolate the style of an artistic image (the style image) and blend or adapt it to the content image to produce a new image called Target image.

How do we Isolate Contents and Styles?

This piece would describe the method used in the 2014 CVPR paper titled: “Image Style Transfer Using Convolutional Neural Networks”

VGG19 architecture (Source)

On close observation of the network above, it can be seen that it is made up of 5 stacks of 2 or 4 convolution layers separated by a maximum pooling layer. The depth of the layers in each stack is a standard value but the depth increases stack after stack from 64 to 512. The network typically accepts a coloured image. In our case, the coloured images would be the content image, style image and target image. The paper makes use of the feature response from the conv4_2 layer. Any of the deeper layers could have been used and still obtain some good results. One of the objectives of the algorithm is to minimize the difference between the content representation and the actual new artistic image we want to produce. That is the newly styled image (target representation) and the original content image would be as close to each other and the network would almost regard them as the same image. The loss is represented as the mean difference of the content representation and the target representation.

Since we are using a pre-trained VGG19 network, we do not need to do any training that would allow us to extract the features. We only need to feed the content and target image into the network optimizing such that the mean difference between them is close to each other as much as possible. It should be noted that the content representation of our content image does not change. However, the content representation of the target image would continue to change until the content loss is minimized.

To represent the style of an image is a bit trickier but equally simple like the previously discussed content representation.

The style representation is found by calculating the correlation (similarity) at each layer? The correlation at each layer is given by a Gram Matrix. Let us remember that the job of the maximum pooling layer is to discard the styles at each successive layer. Thus, we take the feature map just before the discarding occur.

How to find the Gram Matrix

Let us take for an example that we have an 8*8 image convolved with 20 feature maps or kernels. The resultant output would be 20*8*8. Which means we have 20 feature map we want to find the correlation between them. To find this correlation, we first of need to vectorized the values in this feature map (more or less of reshaping the 20*8*8 the 3 dimensions to 2 dimensions). Since we have 20 8*8 vectors, we flatten the x and y dimension of the 8*8 image to give us a length of 1*64 which now represent a feature map. Stacking the 20 rows of 1*64 feature map would give us a 20*64 feature map. The gram matrix is now given by multiplying the 20*64 feature map with its transpose to give us a 20*20 matrix which is non-localized information of our feature map that are unrelated in the orthogonal space. From the above, we can see that the dimension of the Gram matrix is simply a square matric with a dimension of the number of feature maps and does not depend on the dimension of the input image.

Row x and column y of the Gram Matrix simply represent the correlation between the xth and yth feature map in a layer. From this, the diagonal of the gram matrix would simply have a correlation value of 1 since we are finding the similarity between a feature map and itself in this case.

The style loss is simply given as the sum of the mean squared difference between the gram matrix of out style image and the target image. The objective is to minimize this difference. Again, the gram matrix of the style image which represents the style representation of the style image does not change. However, the style representation of the target image would continue to change until the objective is minimized.

The weight w in each layer are values that give some preference (more or less weight depending on the layer)

The total loss function is given above as the sum of the weight of the style loss and the content loss. However, the two losses are calculated differently and have different magnitude. Thus, there is a possibility that the content loss would impact our target image more and we would like to make sure that the two losses impact the final target image almost equally. Alpha and beta are introduced in the original paper as weights used to make the total weight show a balance between the content loss and style loss. Beta in practice always has a higher magnitude than alpha. The lower the alpha-beta ratio, the more stylistic our final target image is going to be as it means that beta in such case has a very much higher magnitude. Alpha and beta are thus hyperparameters we tweak and choose depending on the kind of image we have and how much stylized we want our resulting output to be.

For the example given in this work, alpha is chosen as 1 and beta as 0.00001.

Initializing the target image

Once again, it should be noted that the target image is the new artistic image we want to produce. We could use any random noise value to initialize the image or make it a constant colour image or black image with rows and columns of zeros or white image with rows and columns of ones. In this work, the target image has been initialized with a copy of the content image.

Some Results

The results below show samples of the optimization process as we attempt to achieve the minimization objective. We would see that the style is being gradually learnt and adapted on the content image as shown in the images below.

The final styled image is shown below:

The code used for this work can be found here