# The Intra-Class and Inter-Class Relationships in Style Transfer
## 1 Introduction  
Transferring the look and feel (style) of one image to another image is an interesting but difficult problem. Many researchers are committed to the study of the automatic style transfer, and have developed many efficient and excellent methods [1; 2; 3]. Recently, [PERSON] et al. proposed a pioneering work which captures the artistic style of the images and transfers it to other images by using the convolutional neural network (CNN) [4]. This work formulated the style transfer problem as trying to create an image which matches both the content and style statistics for a given pair of images which provide content and style information separately based on the neural activations of each layer in CNN. Although this work has attracted lots of attention and demonstrated interesting and remarkable visual results in both research laboratory and industry, the key technology of the work, especially the reason that _Gram matrix_ can represent artistic style, remains a mystery.  
Recently, [PERSON] et al. demystified the Neural Style Transfer algorithm by theoretically showing that matching Gram matrices in neural style transfer can be reformulated as minimizing Maximum Mean Discrepancy (MMD) with the second order polynomial kernel [5]. The authors argued that the _Gram matrix_ is not a necessary reference for style transfer, and the neural style transfer is actually a process of distribution alignment of the neural activations between images. Even though this work unlocksthe secret of the _Gram matrix_ to a certain extent, no one has studied the relationships between feature maps obtained from the CNN for the statistical distributions of feature maps.  
Here, we define the feature map for the same layer obtained from the CNN as a feature map _class_. The relationship between the feature map and itself (or its variants) is called _intra-class relationship_. All computation performed on a particular feature map without the effect of other feature maps are called _intra-class operation_. For example, for a particular feature map \(x\), computing the mean and standard deviation of \(x\), or multiplying \(x\) with itself, are all intra-class operations. Similarly, the _inter-class relationship_ is defined for the relationship between different feature maps in the same layer, and the _inter-class operation_ is defined as the computations performed on different feature maps.  
We analyzed the methods proposed in [4; 5], and discovered that in the former method, both intra-class and inter-class relationships are considered, while, in the latter, only the intra-class relationships are considered. According to our experimental results, we argue that the latter produces finer strokes and keeps more content details in the generated images. Furthermore, in this paper, we propose two new methods, called _Cov-Matrix_ and _Cov-MDE-Matrix_ algorithms, to perform style transfer. The former considers both intra-class and inter-class relationships, while the latter only considers the intra-class relationships. We compared our algorithms with the algorithms mentioned above, achieved appealing results and observed a similar trend as mentioned above. We believe that the properties we observed can help researchers to design style transfer methods for different visual demands.  
The rest of this paper is organized as follows: Section 2 introduces some background and related work of style transfer. In Section 3, we analyze the intra-class and inter-class relationships in [4; 5], and propose our argument. Furthermore, two new methods of style transfer are proposed to verify the argument we have presented. The experimental results are shown in Section 4, which verify our argument proposed in Section 3, and show our methods can generate comparable visual effect yet with a lower complexity. Section 5 summarizes our work.