Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph

1 INTRODUCTION . Human perceptual systems routinely separate content and style to better understand their observations ( Tenenbaum & Freeman , 2000 ) . In artificial intelligence , a content and style decomposed representation is also very much desired . However , we notice that existing work does not have a unified definition for content and style . Some definitions are dataset-dependent ( Chou & Lee , 2019 ; Ren et al. , 2021 ) , while some others have to be defined on a certain modality ( Lorenz et al. , 2019 ; Wu et al. , 2019 ) . We wonder , since content-style separation is helpful to our entire perception system , why is there not a unified definition that applies to all perception data ? In order to answer this question , we must first study the characteristics of data . The data of interest , including text , speech , image , and video , are structured . They can be divided into standardized tokens , either naturally as words in language and speech or intentionally as patches in images . Notably , the order of these tokens matters . Disrupting the order of words can make a speech express a completely different meaning . Reversing the order of frames in a video can make a stand-up action become a sit-down action . But there is also some information that is not affected by the order of tokens . For example , scrambling the words in a speech does not change the speaker ’ s voice , and a frame-shuffled video does not change how the person in the video looks . We notice that the content we intuitively think of is affected by the order of tokens , while the style is usually not . Therefore , we could generally define style as token-level permutation invariant ( P.I . ) information and define content as the rest of the information in structured data . However , merely dividing data into two parts is not enough . As Bengio et al . ( 2013 ) pointed out , if we are to take the notion of disentangling seriously , we require a richer interaction of features than that offered by simple linear combinations . An intuitive example is that we could never generate a colored pattern by linearly combining a generic color feature with a gray-scale stimulus pattern . Inspired by this , we propose to model content and style by a token-level bipartite graph , as Figure 1 illustrates . This representation includes an array of content tokens , a set of style tokens , and a set of links modeling the interaction between them . Such a representation allows for fine-grained access and manipulation of features , and enables exciting downstream tasks such as part-level style transfer . In this paper , we design a modal-agnostic framework , named Retriever , for learning bipartite graph representation of content and style . Retriever adopts the autoencoder architecture and addresses two main challenges in it . One is how to decompose the defined content and style in an unsupervised setting . The other is how to compose these two separated factors to reconstruct data . To tackle the first challenge , we employ a cross-attention module that takes the dataset-shared prototype as query to retrieve the style tokens from input data ( Carion et al. , 2020 ) . A cross-attention operation only allows the P.I . information to pass ( Lee et al. , 2019 ) , which is exactly what we want for style . On the other content path , we employ a vector quantization ( VQ ) module ( van den Oord et al. , 2017 ) as the information bottleneck . In addition , we enforce man-induced constraints to make the content tokens interpretable . To tackle the second challenge , we innovate the link attention module for the reconstruction from the bipartite graph . Specifically , the content and style serve as the query and value , respectively . Links between the content and style are learnt and stored in the linking keys . Link attention allows us to retrieve style by content query . As such , the interpretability is propagated from content to style , and the entire representation is friendly to fine-grained editing . We evaluate Retriever in both speech and image domains . In the speech domain , we achieve state-of-the-art ( SOTA ) performance in zero-shot voice conversion , demonstrating a complete and precise decomposition of content and style . In the image domain , we achieve competitive results in part discovery task , which demonstrates the interpretability of the decomposed content . More excitingly , we try part-level style transfer , which can not be offered by most of the existing contentstyle disentanglement approaches . Vivid and interpretable results are achieved . To summarize , our main contributions are three-folds : i ) We provide an intuitive and modalagnostic definition of content and style for structured data . We are the first to model content and style with a token-level bipartite graph . ii ) We propose an unsupervised framework , named Retriever , for learning the proposed content-style representation . A novel link attention module is designed for data reconstruction from content-style bipartite graph . iii ) We demonstrate the power of Retriever in challenging downstream tasks in both speech and image domains . 2 RELATED WORK . Content-style decomposed representation can be approached in supervised or unsupervised settings . When style labels , such as the speaker labels of speeches ( Kameoka et al. , 2018 ; Qian et al. , 2019 ; Yuan et al. , 2021 ) and the identity labels of face images ( Mathieu et al. , 2016 ; Szabó et al. , 2018 ; Jha et al. , 2018 ; Bouchacourt et al. , 2018 ; Gabbay & Hoshen , 2020 ) , are available , latent variables can be divided into content and style based on group supervision . Recently , there has been increased interest in unsupervised learning of content and style . Since there is no explicit supervision signal , the basic problem one must first solve is the definition of content and style . We discover that all existing definitions are either domain-specific or task-specific . For example , in speech domain , Chou & Lee ( 2019 ) assume that style is the global statistical information and content is what is left after instance normalization ( IN ) . Ebbers et al . ( 2021 ) suggest that style captures long-term stability and content captures short-term variations . In image domain , the definition is even more diverse . Lorenz et al . ( 2019 ) try to discover the invariants under spatial and appearance transformations and treat them as style and content , respectively . Wu et al . ( 2019 ) define content as 2D landmarks and style as the rest of information . Ren et al . ( 2021 ) define content as the most important factor across the whole dataset for image reconstruction , which is rather abstract . In this work , we attempt to find a general and modal-agnostic definition of content and style . Style transfer is partially related to our work , as it concerns the combination of content and style to reconstruct data . AdaIN ( Huang & Belongie , 2017 ; Chou & Lee , 2019 ) goes beyond the linear combination of content and style ( Tenenbaum & Freeman , 2000 ) and proposes to inject style into content by aligning the mean and variance of the content features with those of the style features . However , style is not separated from content in this line of research . Liu et al . ( 2021 ) touch upon the part-based style transfer task as we do . They model the relationship between content and style by a one-one mapping . They follow the common definition of content and style in the image domain as shape and appearance , and try to disentangle them with hand-crafted data augmentation methods . Besides , the term “ style ” is often seen in image generative models , such as StyleGAN ( Karras et al. , 2019 ) . However , the style mentioned in this type of work is conceptually different from the style in our work . In StyleGAN , there is no concept of content , and style is the whole latent variable containing all the information including the appearance and shape of an image . Following StyleGAN , Hudson & Zitnick ( 2021 ) employ a bipartite structure to enable long-range interactions across the image , which iteratively propagates information from a set of latent variables to the evolving visual features . Recently , researchers have become interested in disentangling content and style from the latent variables of StyleGAN ( Alharbi & Wonka , 2020 ; Kwon & Ye , 2021 ) . However , they only work for well-aligned images and are hard to be applied to other modalities . 3 CONTENT-STYLE REPRESENTATION FOR STRUCTURED DATA . In this section , we provide definitions of content and style for structured data , introduce the framework for content-style decomposition , and propose the token-level bipartite graph representation . 3.1 DEFINITION OF CONTENT AND STYLE . The data of interest is structured data that can be tokenized , denoted by X = [ x1 , x2 , ... , xn ] . We think of text , speech , and image , among many others , as structured data . Each token xi can be a word in text , a phoneme in a speech , or a patch in an image . These data are structured because non-trivial instances ( examples of trivial instances are a silent speech or a blank image ) are not able to keep their full information when the order of tokens is not given . Inspired by this intuition , we define style of X as the information that is not affected by the permutation of tokens , or permutation invariant ( P.I . ) information . Content is the rest of information in X . 3.2 CONTENT-STYLE SEPARATION . The information in a piece of structured data is carried either in the content or in the style . By definition , style can be extracted by a P.I . function fP.I . ( · ) , which satisfies f ( π ( X ) ) = f ( X ) , where π ( · ) represents permutation of tokens . To achieve content-style decomposition , we naturally adopt an autoencoder architecture , as shown in Figure 2 . The bottom path is the style path which implements fP.I. ( · ) . We shall find a powerful P.I . function which will let all the P.I . information pass . The top path is responsible for extracting the content , but the challenge is that there does not exist a function which only lets pass the non-P.I . information . Therefore , we employ in the content path a permutation-variant function , which will let all information pass , including the P.I . information . Obviously , if we do not pose any constraint on the content path , as shown in Figure 2 ( a ) , the style information will be leaked to the content path . To squeeze the style information out of the content path , an information bottleneck fB.N . ( · ) is required . A perfect bottleneck , as shown in Figure 2 ( c ) , can avoid the style leak while achieving perfect reconstruction . But an imperfect bottleneck , being too wide or too narrow , will cause style leak or content loss , as shown in Figure 2 ( b ) and Figure 2 ( d ) , respectively .

Summary: The paper proposes a novel idea of disentangling the content and style of structured data by treating style as permutation invariant information. It adopts VQ network for content encoding, and Cross-Attention for Style and Linking Attention at decoder. It is shown to be domain agonistic - worked well in image and audio domain.