Open-Ended Content-Style Recombination Via Leakage Filtering

We consider visual domains in which a class label specifies the content of an image, and class-irrelevant properties that differentiate instances constitute the style. We present a domain-independent method that permits the open-ended recombination of style of one image with the content of another. Open ended simply means that the method generalizes to style and content not present in the training data. The method starts by constructing a content embedding using an existing deep metric-learning technique. This trained content encoder is incorporated into a variational autoencoder (VAE), paired with a to-be-trained style encoder. The VAE reconstruction loss alone is inadequate to ensure a decomposition of the latent representation into style and content. Our method thus includes an auxiliary loss, leakage filtering, which ensures that no style information remaining in the content representation is used for reconstruction and vice versa. We synthesize novel images by decoding the style representation obtained from one image with the content representation from another. Using this method for data-set augmentation, we obtain state-of-the-art performance on few-shot learning tasks. In any domain involving classification, entities are distinguished not only by class label but also by attributes orthogonal to class label. For example, if faces are classified by identity, within-class variation is due to lighting, pose, expression, hairstyle; if masterworks of art are classified by the painter, within-class variation is due to choice of subject matter. Following tradition (Tenenbaum & Freeman, 2000), we refer to betweenand within-class variation as content and style, respectively. What constitutes content is defined with respect to a task. For example, in a face-recognition task, identity is the content; in an emotion-recognition task, expression is the content. There has been a wealth of research focused on decomposing content and style, with the promise that decompositions might provide insight into a domain or improve classification performance. Decompositions also allow for the synthesis of novel entities by recombining the content of one entity with the style of another. Recombinations are interesting as a creative exercise (e.g., transforming the musical composition of one artist in the style of another) or for data set augmentation. We propose an approach to content-style decomposition and recombination. We refer to the method as STOC, for Style Transfer onto Open-Ended Content. Our approach is differentiated from past work in the following ways. First, STOC can transfer to novel content. In contrast, most previous work assumes the content classes in testing are the same as those in training. Second, STOC is general purpose and can be applied to any domain. In contrast, previous work includes approaches that leverage specific domain knowledge (e.g., human body pose). Third, STOC has an explicit objective, leakage filtering, designed to isolate content and style. No such explicit objective is found in most previous work, and as a result, synthesized examples may fail to preserve content as style is varied and vice versa. Fourth, STOC requires a labeling of entities by content class, but explicit style labels are not required. In contrast, some previous work assumes supervised training of both style and content representations.  1 shows examples of content-style recombination using STOC on the VGG-Face (Parkhi et al., 2015) data set. In each column, the content of the image in the top row is combined with the style of the image on the bottom row to synthesize a novel image, shown in the middle row. The images in the top and bottom rows are of identities (content) held out from the training set. Style is well 1 ar X iv :1 81 0. 00 11 0v 1 [ cs .L G ] 2 8 Se p 20 18 test image used to infer content test image used to infer style recombined image  Examples of content-style recombination using STOC on the VGG-Face data set. The middle face in each column combines content of the top face with style of the bottom face. maintained, and content is fairly well transferred, at least to the degree that the faces in the middle row are more similar to top-row than bottom-row faces. The training faces are labeled by identity, but style is induced by the training procedure. 1 PAST RESEARCH ON STYLE TRANSFER A growing body of work has demonstrated impressive style transfer with models that can translate images from one specific domain (or content class) to another. Although some of these approaches require paired samples from both domains (Isola et al., 2017), CycleGAN and its variants does not (Zhu et al., 2017; Choi et al., 2018). This method has been extended to exploit constraints in video (Bansal et al., 2018), yielding impressive sequences in which the mannerisms and facial movements of one individual are transferred to another. These methods are dependent on having many examples from pairs of content classes, and a model is custom trained for that pair. Therefore, the models do not attempt to learn an explicit representation of content or to decompose style and content. Some domain-to-domain translation models do perform disentangling—of the information in an entity that is shared between domains and the information that is not shared (Gonzalez-Garcia et al., 2018; Huang et al., 2018). Huang et al. (2018) refer to this as content-style decomposition, but the range of content is quite restricted. For example, a model might be trained to transform cats into lions, but it cannot subsequently be used to transform cats into, say, panthers. An early proposal for style transfer (Kingma et al., 2014), based on variational autoencoders, can translate between more than two domains, but the model is unable to handle novel domains in the test set. An open-ended method such as STOC can process novel content. Many of the above techniques are described as unsupervised because no cross-domain correspondence between examples is required. However, from our perspective, the separation of examples by domain is a form of supervision, the same form we leverage in STOC. Previous techniques that allow for open-ended content have typically required supervisory signals for both content and style. That is, labels must be provided for the content class of each training sample as well as for each of a specified set of style dimensions such as pose and lighting (Karaletsos et al., 2016; Kulkarni et al., 2015; Reed et al., 2014). Analogy constraints of the form x1 : x2 :: y1 : y2 have also been explored as a supervisory signal for style, specifying that two samples of one class X have the same stylistic variation as two samples of another class, Y (Reed et al., 2015). Methods have been developed that can transfer style to novel content without requiring explicit style labels but instead rely on domain-specific knowledge. For example, Jetchev & Bergmann (2017) demonstrate the transfer of novel articles of clothing onto novel individuals, but their approach assumes that style transfer can be applied to only a masked region of the image. Other work has leveraged constraints inherent in a video sequence, either in a strong manner by extracting pose from the video (Brand & Hertzmann, 2000; Chan et al., 2018; Hsu et al., 2005), or in a weaker fashion by decomposing a video sequence into stationary (content) and nonstationary (style) components (Denton & Birodkar, 2017; Tulyakov et al., 2018).