Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph

1 INTRODUCTION . Human perceptual systems routinely separate content and style to better understand their observations ( Tenenbaum & Freeman , 2000 ) . In artificial intelligence , a content and style decomposed representation is also very much desired . However , we notice that existing work does not have a unified definition for content and style . Some definitions are dataset-dependent ( Chou & Lee , 2019 ; Ren et al. , 2021 ) , while some others have to be defined on a certain modality ( Lorenz et al. , 2019 ; Wu et al. , 2019 ) . We wonder , since content-style separation is helpful to our entire perception system , why is there not a unified definition that applies to all perception data ? In order to answer this question , we must first study the characteristics of data . The data of interest , including text , speech , image , and video , are structured . They can be divided into standardized tokens , either naturally as words in language and speech or intentionally as patches in images . Notably , the order of these tokens matters . Disrupting the order of words can make a speech express a completely different meaning . Reversing the order of frames in a video can make a stand-up action become a sit-down action . But there is also some information that is not affected by the order of tokens . For example , scrambling the words in a speech does not change the speaker ’ s voice , and a frame-shuffled video does not change how the person in the video looks . We notice that the content we intuitively think of is affected by the order of tokens , while the style is usually not . Therefore , we could generally define style as token-level permutation invariant ( P.I . ) information and define content as the rest of the information in structured data . However , merely dividing data into two parts is not enough . As Bengio et al . ( 2013 ) pointed out , if we are to take the notion of disentangling seriously , we require a richer interaction of features than that offered by simple linear combinations . An intuitive example is that we could never generate a colored pattern by linearly combining a generic color feature with a gray-scale stimulus pattern . Inspired by this , we propose to model content and style by a token-level bipartite graph , as Figure 1 illustrates . This representation includes an array of content tokens , a set of style tokens , and a set of links modeling the interaction between them . Such a representation allows for fine-grained access and manipulation of features , and enables exciting downstream tasks such as part-level style transfer . In this paper , we design a modal-agnostic framework , named Retriever , for learning bipartite graph representation of content and style . Retriever adopts the autoencoder architecture and addresses two main challenges in it . One is how to decompose the defined content and style in an unsupervised setting . The other is how to compose these two separated factors to reconstruct data . To tackle the first challenge , we employ a cross-attention module that takes the dataset-shared prototype as query to retrieve the style tokens from input data ( Carion et al. , 2020 ) . A cross-attention operation only allows the P.I . information to pass ( Lee et al. , 2019 ) , which is exactly what we want for style . On the other content path , we employ a vector quantization ( VQ ) module ( van den Oord et al. , 2017 ) as the information bottleneck . In addition , we enforce man-induced constraints to make the content tokens interpretable . To tackle the second challenge , we innovate the link attention module for the reconstruction from the bipartite graph . Specifically , the content and style serve as the query and value , respectively . Links between the content and style are learnt and stored in the linking keys . Link attention allows us to retrieve style by content query . As such , the interpretability is propagated from content to style , and the entire representation is friendly to fine-grained editing . We evaluate Retriever in both speech and image domains . In the speech domain , we achieve state-of-the-art ( SOTA ) performance in zero-shot voice conversion , demonstrating a complete and precise decomposition of content and style . In the image domain , we achieve competitive results in part discovery task , which demonstrates the interpretability of the decomposed content . More excitingly , we try part-level style transfer , which can not be offered by most of the existing contentstyle disentanglement approaches . Vivid and interpretable results are achieved . To summarize , our main contributions are three-folds : i ) We provide an intuitive and modalagnostic definition of content and style for structured data . We are the first to model content and style with a token-level bipartite graph . ii ) We propose an unsupervised framework , named Retriever , for learning the proposed content-style representation . A novel link attention module is designed for data reconstruction from content-style bipartite graph . iii ) We demonstrate the power of Retriever in challenging downstream tasks in both speech and image domains . 2 RELATED WORK . Content-style decomposed representation can be approached in supervised or unsupervised settings . When style labels , such as the speaker labels of speeches ( Kameoka et al. , 2018 ; Qian et al. , 2019 ; Yuan et al. , 2021 ) and the identity labels of face images ( Mathieu et al. , 2016 ; Szabó et al. , 2018 ; Jha et al. , 2018 ; Bouchacourt et al. , 2018 ; Gabbay & Hoshen , 2020 ) , are available , latent variables can be divided into content and style based on group supervision . Recently , there has been increased interest in unsupervised learning of content and style . Since there is no explicit supervision signal , the basic problem one must first solve is the definition of content and style . We discover that all existing definitions are either domain-specific or task-specific . For example , in speech domain , Chou & Lee ( 2019 ) assume that style is the global statistical information and content is what is left after instance normalization ( IN ) . Ebbers et al . ( 2021 ) suggest that style captures long-term stability and content captures short-term variations . In image domain , the definition is even more diverse . Lorenz et al . ( 2019 ) try to discover the invariants under spatial and appearance transformations and treat them as style and content , respectively . Wu et al . ( 2019 ) define content as 2D landmarks and style as the rest of information . Ren et al . ( 2021 ) define content as the most important factor across the whole dataset for image reconstruction , which is rather abstract . In this work , we attempt to find a general and modal-agnostic definition of content and style . Style transfer is partially related to our work , as it concerns the combination of content and style to reconstruct data . AdaIN ( Huang & Belongie , 2017 ; Chou & Lee , 2019 ) goes beyond the linear combination of content and style ( Tenenbaum & Freeman , 2000 ) and proposes to inject style into content by aligning the mean and variance of the content features with those of the style features . However , style is not separated from content in this line of research . Liu et al . ( 2021 ) touch upon the part-based style transfer task as we do . They model the relationship between content and style by a one-one mapping . They follow the common definition of content and style in the image domain as shape and appearance , and try to disentangle them with hand-crafted data augmentation methods . Besides , the term “ style ” is often seen in image generative models , such as StyleGAN ( Karras et al. , 2019 ) . However , the style mentioned in this type of work is conceptually different from the style in our work . In StyleGAN , there is no concept of content , and style is the whole latent variable containing all the information including the appearance and shape of an image . Following StyleGAN , Hudson & Zitnick ( 2021 ) employ a bipartite structure to enable long-range interactions across the image , which iteratively propagates information from a set of latent variables to the evolving visual features . Recently , researchers have become interested in disentangling content and style from the latent variables of StyleGAN ( Alharbi & Wonka , 2020 ; Kwon & Ye , 2021 ) . However , they only work for well-aligned images and are hard to be applied to other modalities . 3 CONTENT-STYLE REPRESENTATION FOR STRUCTURED DATA . In this section , we provide definitions of content and style for structured data , introduce the framework for content-style decomposition , and propose the token-level bipartite graph representation . 3.1 DEFINITION OF CONTENT AND STYLE . The data of interest is structured data that can be tokenized , denoted by X = [ x1 , x2 , ... , xn ] . We think of text , speech , and image , among many others , as structured data . Each token xi can be a word in text , a phoneme in a speech , or a patch in an image . These data are structured because non-trivial instances ( examples of trivial instances are a silent speech or a blank image ) are not able to keep their full information when the order of tokens is not given . Inspired by this intuition , we define style of X as the information that is not affected by the permutation of tokens , or permutation invariant ( P.I . ) information . Content is the rest of information in X . 3.2 CONTENT-STYLE SEPARATION . The information in a piece of structured data is carried either in the content or in the style . By definition , style can be extracted by a P.I . function fP.I . ( · ) , which satisfies f ( π ( X ) ) = f ( X ) , where π ( · ) represents permutation of tokens . To achieve content-style decomposition , we naturally adopt an autoencoder architecture , as shown in Figure 2 . The bottom path is the style path which implements fP.I. ( · ) . We shall find a powerful P.I . function which will let all the P.I . information pass . The top path is responsible for extracting the content , but the challenge is that there does not exist a function which only lets pass the non-P.I . information . Therefore , we employ in the content path a permutation-variant function , which will let all information pass , including the P.I . information . Obviously , if we do not pose any constraint on the content path , as shown in Figure 2 ( a ) , the style information will be leaked to the content path . To squeeze the style information out of the content path , an information bottleneck fB.N . ( · ) is required . A perfect bottleneck , as shown in Figure 2 ( c ) , can avoid the style leak while achieving perfect reconstruction . But an imperfect bottleneck , being too wide or too narrow , will cause style leak or content loss , as shown in Figure 2 ( b ) and Figure 2 ( d ) , respectively .

Summary: This paper proposes an unsupervised framework to learn content and style representations of structured data, such as texts, images and speeches. The task is interesting to me and decomposition representation learning is important. In addition, a bipartite graph is proposed to model the link between content tokens and style tokens, making the model more interpretable. The proposed model performs relatively well on both speech and image domains.