Year: 2021

Abstract:
This paper addresses the unsupervised learning of content-style decomposed representation. We first give a definition of style and then model the content-style representation as a token-level bipartite graph. An unsupervised framework, named Retriever, is proposed to learn such representations. First, a cross-attention module is employed to retrieve permutation invariant (P.I.) information, defined as style, from the input data. Second, a vector quantization (VQ) module is used, together with man-induced constraints, to produce interpretable content tokens. Last, an innovative link attention module serves as the decoder to reconstruct data from the decomposed content and style, with the help of the linking keys. Being modal-agnostic, the proposed Retriever is evaluated in both speech and image domains. The state-of-the-art zero-shot voice conversion performance confirms the disentangling ability of our framework. Top performance is also achieved in the part discovery task for images, verifying the interpretability of our representation. In addition, the vivid part-based style transfer quality demonstrates the potential of Retriever to support various fascinating generative tasks. Code will be released upon acceptance of the paper.

heading: 1 INTRODUCTION
text: Human perceptual systems routinely separate content and style to better understand their observations (Tenenbaum & Freeman, 2000). In artificial intelligence, a content and style decomposed representation is also very much desired. However, we notice that existing work does not have a unified definition for content and style. Some definitions are dataset-dependent (Chou & Lee, 2019; Ren et al., 2021), while some others have to be defined on a certain modality (Lorenz et al., 2019; Wu et al., 2019). We wonder, since content-style separation is helpful to our entire perception system, why is there not a unified definition that applies to all perception data?
In order to answer this question, we must first study the characteristics of data. The data of interest, including text, speech, image, and video, are structured. They can be divided into standardized tokens, either naturally as words in language and speech or intentionally as patches in images. Notably, the order of these tokens matters. Disrupting the order of words can make a speech express a completely different meaning. Reversing the order of frames in a video can make a stand-up action become a sit-down action. But there is also some information that is not affected by the order of tokens. For example, scrambling the words in a speech does not change the speaker’s voice, and a frame-shuffled video does not change how the person in the video looks. We notice that the content we intuitively think of is affected by the order of tokens, while the style is usually not. Therefore, we could generally define style as token-level permutation invariant (P.I.) information and define content as the rest of the information in structured data.
However, merely dividing data into two parts is not enough. As Bengio et al. (2013) pointed out, if we are to take the notion of disentangling seriously, we require a richer interaction of features than that offered by simple linear combinations. An intuitive example is that we could never generate a colored pattern by linearly combining a generic color feature with a gray-scale stimulus pattern. Inspired by this, we propose to model content and style by a token-level bipartite graph, as Figure 1 illustrates. This representation includes an array of content tokens, a set of style tokens, and a set of links modeling the interaction between them. Such a representation allows for fine-grained access and manipulation of features, and enables exciting downstream tasks such as part-level style transfer.
In this paper, we design a modal-agnostic framework, named Retriever, for learning bipartite graph representation of content and style. Retriever adopts the autoencoder architecture and addresses two main challenges in it. One is how to decompose the defined content and style in an unsupervised setting. The other is how to compose these two separated factors to reconstruct data.
To tackle the first challenge, we employ a cross-attention module that takes the dataset-shared prototype as query to retrieve the style tokens from input data (Carion et al., 2020). A cross-attention operation only allows the P.I. information to pass (Lee et al., 2019), which is exactly what we want for style. On the other content path, we employ a vector quantization (VQ) module (van den Oord et al., 2017) as the information bottleneck. In addition, we enforce man-induced constraints to make the content tokens interpretable. To tackle the second challenge, we innovate the link attention module for the reconstruction from the bipartite graph. Specifically, the content and style serve as the query and value, respectively. Links between the content and style are learnt and stored in the linking keys. Link attention allows us to retrieve style by content query. As such, the interpretability is propagated from content to style, and the entire representation is friendly to fine-grained editing.
We evaluate Retriever in both speech and image domains. In the speech domain, we achieve state-of-the-art (SOTA) performance in zero-shot voice conversion, demonstrating a complete and precise decomposition of content and style. In the image domain, we achieve competitive results in part discovery task, which demonstrates the interpretability of the decomposed content. More excitingly, we try part-level style transfer, which cannot be offered by most of the existing contentstyle disentanglement approaches. Vivid and interpretable results are achieved.
To summarize, our main contributions are three-folds: i) We provide an intuitive and modalagnostic definition of content and style for structured data. We are the first to model content and style with a token-level bipartite graph. ii) We propose an unsupervised framework, named Retriever, for learning the proposed content-style representation. A novel link attention module is designed for data reconstruction from content-style bipartite graph. iii) We demonstrate the power of Retriever in challenging downstream tasks in both speech and image domains.

heading: 2 RELATED WORK
text: Content-style decomposed representation can be approached in supervised or unsupervised settings. When style labels, such as the speaker labels of speeches (Kameoka et al., 2018; Qian et al., 2019; Yuan et al., 2021) and the identity labels of face images (Mathieu et al., 2016; Szabó et al., 2018; Jha et al., 2018; Bouchacourt et al., 2018; Gabbay & Hoshen, 2020), are available, latent variables can be divided into content and style based on group supervision.
Recently, there has been increased interest in unsupervised learning of content and style. Since there is no explicit supervision signal, the basic problem one must first solve is the definition of content and style. We discover that all existing definitions are either domain-specific or task-specific. For example, in speech domain, Chou & Lee (2019) assume that style is the global statistical information and content is what is left after instance normalization (IN). Ebbers et al. (2021) suggest that style captures long-term stability and content captures short-term variations. In image domain, the definition is even more diverse. Lorenz et al. (2019) try to discover the invariants under spatial and appearance transformations and treat them as style and content, respectively. Wu et al. (2019) define
content as 2D landmarks and style as the rest of information. Ren et al. (2021) define content as the most important factor across the whole dataset for image reconstruction, which is rather abstract. In this work, we attempt to find a general and modal-agnostic definition of content and style.
Style transfer is partially related to our work, as it concerns the combination of content and style to reconstruct data. AdaIN (Huang & Belongie, 2017; Chou & Lee, 2019) goes beyond the linear combination of content and style (Tenenbaum & Freeman, 2000) and proposes to inject style into content by aligning the mean and variance of the content features with those of the style features. However, style is not separated from content in this line of research. Liu et al. (2021) touch upon the part-based style transfer task as we do. They model the relationship between content and style by a one-one mapping. They follow the common definition of content and style in the image domain as shape and appearance, and try to disentangle them with hand-crafted data augmentation methods.
Besides, the term “style” is often seen in image generative models, such as StyleGAN (Karras et al., 2019). However, the style mentioned in this type of work is conceptually different from the style in our work. In StyleGAN, there is no concept of content, and style is the whole latent variable containing all the information including the appearance and shape of an image. Following StyleGAN, Hudson & Zitnick (2021) employ a bipartite structure to enable long-range interactions across the image, which iteratively propagates information from a set of latent variables to the evolving visual features. Recently, researchers have become interested in disentangling content and style from the latent variables of StyleGAN (Alharbi & Wonka, 2020; Kwon & Ye, 2021). However, they only work for well-aligned images and are hard to be applied to other modalities.

heading: 3 CONTENT-STYLE REPRESENTATION FOR STRUCTURED DATA
text: In this section, we provide definitions of content and style for structured data, introduce the framework for content-style decomposition, and propose the token-level bipartite graph representation.

heading: 3.1 DEFINITION OF CONTENT AND STYLE
text: The data of interest is structured data that can be tokenized, denoted by X = [x1,x2, ...,xn]. We think of text, speech, and image, among many others, as structured data. Each token xi can be a word in text, a phoneme in a speech, or a patch in an image. These data are structured because non-trivial instances (examples of trivial instances are a silent speech or a blank image) are not able to keep their full information when the order of tokens is not given. Inspired by this intuition, we define style of X as the information that is not affected by the permutation of tokens, or permutation invariant (P.I.) information. Content is the rest of information in X .

heading: 3.2 CONTENT-STYLE SEPARATION
text: The information in a piece of structured data is carried either in the content or in the style. By definition, style can be extracted by a P.I. function fP.I.(·), which satisfies f(π(X)) = f(X), where π(·) represents permutation of tokens. To achieve content-style decomposition, we naturally adopt an autoencoder architecture, as shown in Figure 2. The bottom path is the style path which implements fP.I.(·). We shall find a powerful P.I. function which will let all the P.I. information pass. The top path is responsible for extracting the content, but the challenge is that there does not exist a function which only lets pass the non-P.I. information. Therefore, we employ in the content path a permutation-variant function, which will let all information pass, including the P.I. information.
Obviously, if we do not pose any constraint on the content path, as shown in Figure 2 (a), the style information will be leaked to the content path. To squeeze the style information out of the content path, an information bottleneck fB.N.(·) is required. A perfect bottleneck, as shown in Figure 2 (c), can avoid the style leak while achieving perfect reconstruction. But an imperfect bottleneck, being too wide or too narrow, will cause style leak or content loss, as shown in Figure 2 (b) and Figure 2 (d), respectively.

heading: 3.3 TOKEN-LEVEL BIPARTITE GRAPH REPRESENTATION OF CONTENT AND STYLE
text: After dividing the structured input X into content feature C and style feature S, we continue to explore how the relationship between C and S should be modeled. Ideally, we do not want an all-to-one mapping, in which the style is applied to all the content as a whole. Nor do we want a one-to-one mapping where each content token is associated with a fixed style. In order to provide a flexible way of interaction, we propose a novel token-level bipartite graph modeling of C and S.
The token-level representations of C and S are C = [c1, c2, ..., cn] and S = {s1, s2, ..., sm}, respectively. Note that there is a one-one correspondence between xi and ci, so the order of the structured data is preserved in the content C. While the order is preserved, the semantic meaning of each ci is not fixed. This suggests that the bipartite graph between C and S cannot be static. In order to model a dynamic bipartite graph, we introduce a set of learnable linking keys K = {k1,k2, ...,km}. The linking keys and style tokens form a set of key-value pairs {(ki, si)}mi=1. Our linking key design allows for a soft and learnable combination of content and style. The connection weight between a content token ci and a style token sj is calculated by wi,j = rθ(ci,kj), where rθ is a learnable linking function parameterized with θ. For a content token ci, its content-specific style feature now can be calculated by sθ(ci) = ∑m j=1 ŵi,jsj , where ŵi,j are normalized weights.
4 THE RETRIEVER FRAMEWORK

heading: 4.1 OVERVIEW
text: This section presents the design and implementation of the Retriever framework, which realizes the bipartite graph representation of content and style as described in Section 3. Retriever, as shown in Figure 3, is an autoencoder comprising of four major blocks, two of which implement the most important style retrieval functions. This is why the name Retriever is coined.
Retriever is modal-agnostic. The input of the framework is an array of input tokens, denoted by X = [x1,x2, ..,xn] ∈ Rn×d, obtained by modal-specific tokenization operations from raw data. The first major component is a pre-processing module to extract the feature F ∈ Rn×d from X . We implement it as a stack (Le) of non-P.I. transformer encoder blocks (Vaswani et al., 2017) to keep all the important information. Then F is decomposed by the content encoder fB.N.(·) and the style encoder fP.I.(·). fB.N.(X) is implemented by vector quantization to extract content C = [c1, c2, ..., cn]. Structural constraint can be imposed on C to improve interpretability. A cross-attention module is employed as fP.I.(X) to retrieve style S = {s1, s2, ..., sm}. At last, the decoder, implemented by the novel link attention module, reconstructs X from the bipartite graph.

heading: 4.2 CONTENT ENCODER
text: Content encoder should serve two purposes. First, it should implement an information bottleneck to squeeze out the style information. Second, it should provide interpretability such that meaningful feature editing can be enabled.
Information bottleneck. Vector quantization (VQ) is a good candidate to implement information bottleneck (Wu et al., 2020), although many other choices can easily fit into our framework as well. VQ maps data from a continuous space into a set of discrete representations with restricted information capacity, and has shown strong generalization ability to multiple data formats, such as image (van den Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021), audio (Baevski et al., 2020a;b) and video (Yan et al., 2021).
We use product quantization (Baevski et al., 2020a;b), which represents the input F with a concatenation ofG groups of codes, and each group of codes is from an independent codebook of V entries. To encourage all the VQ codes to be equally used, we use a batch-level VQ perplexity loss, denoted as LV Q. Please refer to Appendix A for more details. When the bottleneck capacity is appropriate, the content can be extracted as C = VQ(F ), where C ∈ Rn×dc . Interpretability. Priori knowledge can be imposed on the content path of Retriever to increase the interpretability of feature representations. In this work, we introduce structural constraint LSC to demonstrate this capability. Other prior knowledge, including modal-specific ones, can be added to the framework as long as they can be translated into differentiable loss functions.
The structural constraint we have implemented in Retriever is a quite general one, as it reflects the locality bias widely exists in natural signals. In image domain, we force the spatially adjacent tokens to share the same VQ code, so that a single VQ code may represent a meaningful object part. In speech domain, we discourage the VQ code from changing too frequently along the temporal axis. As such, adjacent speech tokens can share the same code that may represent a phoneme. The visualization in the next section will demonstrate the interpretability of Retriever features.

heading: 4.3 STYLE ENCODER WITH CROSS-ATTENTION
text: We have defined style as the P.I. information that can be extracted by a P.I. function from the structured data. Cross-Attention, which is widely used in set prediction task (Carion et al., 2020) and set data processing such as 3D point cloud (Lee et al., 2019), is known to be a P.I. function. It is also a powerful operation that can project a large input to a smaller or arbitrary target shape. Previously, Perceiver (Jaegle et al., 2021b) and Perceiver IO (Jaegle et al., 2021a) have used this operation as a replacement of the self-attention operation in Transformers to reduce the complexity. However, cross-attention operation does not preserve data structure. In order to compensate for this, additional positional embedding is implemented to associate the position information with each input element.
Conversely, our work takes advantage of cross-attention’s non-structure-preserving nature to retrieve the P.I. style information. A key implementation detail is that position information should not be associated with the input tokens. Otherwise, there will be content leaks into the style path. To implement the style encoder, we follow previous work (Carion et al., 2020; Lee et al., 2019) to learn m dataset-shared prototypes Z0 ∈ Rm×ds as seed vectors. Figure 4 (a) shows the implementation details of the style encoder, which is a stack of Ls cross-attention blocks, token mixing layers, and feed-forward network (FFN) layers. The final output is the retrieved style S ∈ Rm×ds .

heading: 4.4 DECODER WITH LINK ATTENTION
text: In an autoencoder architecture, the decoder is responsible for reconstructing data from latent features. In learning a content-style representation, the ability of the decoder limits the way content and style can be modeled. Therefore, decoder design becomes the key to the entire framework design.
In order to support the bipartite graph representation of content and style, we innovatively design link attention by unleashing the full potential of multi-head attention. Unlike self-attention and cross-attention, link attention has distinct input for Q, K, and V . In the Retriever decoder, content tokens are queries and style tokens are values. The keys are a set of learnable linking keys K ∈ Rm×ds which are paired with the style tokens to represent the links in the content-style bipartite graph. Such a design allows us to retrieve content-specific style for data reconstruction, enabling fine-grained feature editing and interpretable style transfer.
Figure 4 (b) shows the implementation details of the Retriever decoder. We first use a convolutional layer directly after VQ to aggregate the contextual information as Y0 = Conv(C). Then the style is injected into content for Ld times. In each round of style injection, we add a self-attention layer before the link attention layer and append an FFN layer after it. The element order in K and S does not matter, as long as ki and si are paired. The output of the decoder is then detokenized to reconstruct the raw data, whose difference with the input raw data is used as the reconstruction loss.

heading: 4.5 LOSS FUNCTIONS
text: Our final loss function Lsum consists of three components: the reconstruction loss Lrec, the VQ diversity loss LV Q, and the structural constraint loss LSC (implementation details described in Appendix D). It is computed by:
Lsum = λrecLrec + λV QLV Q + λSCLSC , (1)
where λrec, λV Q, and λSC are hyper-parameters controlling the weights of the three losses.

heading: 5 EXPERIMENTS
text: We evaluate the Retriever framework in both speech and image domains. Due to the differences in tasks, datasets, and evaluation metrics, we organize the experiments in these two domains in two subsections. We use visualizations or quantitative results to: i) demonstrate the effect of contentstyle separation and the interpretability of content; ii) illustrate how the bipartite graph representation conveys the interpretability of content to style and supports fine-grained style transfer.

heading: 5.1 SPEECH DOMAIN
text: 5.1.1 TRAINING RETRIEVER FOR SPEECH SIGNALS
Retriever for speech signals is trained with the entire 44-hour CSTR VCTK Corpus (Veaux et al., 2017) containing 109 speakers. We use the pre-trained CPC network (Rivière et al., 2020) followed by a depth-wise convolution to perform tokenization. In Retriever, the content encoder is implemented by a VQ with two groups (G = 2) and V = 100. A truncated neighborhood crossentropy loss is used as LSC to enforce the structural constraint, which is only applied to group #0. For the style encoder, we set the number of style tokens to 60. At the decoder, the content is first processed by a 1D depth-wise convolution with kernel size 31 before being passed to link attention. In the detokenization step, we resample the output log-Mel spectrum to 80Hz and feed it into Parallel WaveGAN vocoder (Yamamoto et al., 2020) to generate the waveform. We apply L1 reconstruction loss on log-Mel spectrum as Lrec. More details can be found in Appendix F.1.

heading: 5.1.2 VISUALIZATION OF CONTENT-STYLE REPRESENTATION
text: To understand how Retriever models the content, the style, and the bipartite graph between them, we visualize the VQ codes and the decoder link attention map for a 2.5s-long test utterance in Figure 5 (a). Each dot represents one token and the token hop is 10ms. Retriever encodes content with VQ, and we further impose structural constraint to VQ Group #0. We find that this group
Table 1: Phoneme information probing.
Settings Frame Frame w/ context
CPC + k-means 9.6% 9.5% Group #0 48.1% 62.7% Group #1 21.0% 45.9% Group #0, #1 52.3% 66.9%
of code demonstrates good interpretability as desired. Codes for adjacent tokens are highly consistent, and the changes of codes are well aligned with ground-truth phoneme changes. The VQ codes of group #1 exhibit a different pattern. They switch more frequently as they capture more detailed residual variations within each phoneme. We further study the interpretability of content by phoneme information probing (implementation described in Appendix F.3), which is used to show the information captured by the discovered content units (Chorowski et al., 2019). We measure the frame-level phoneme classification accuracy with or without context and compare with the baseline (CPC + k-means). Results in Table 1 show that the two groups of VQ codes discovered in our unsupervised framework align surprisingly well with the annotated phoneme labels. Notably, group #0 codes capture most of the phoneme information and group #1 codes are less interpretable. This demonstrates how the structural constraint helps to increase the interpretability of content.
In decoder link attention, all the content tokens within the same phoneme tend to attend to the same style vector, while tokens from different phonemes always attend to different style vectors. We further visualize the co-occurrence map between the ground-truth phonemes and the style vectors in Figure 5 (b) (calculation described in Appendix B). We see that the phonetic content and style vectors have a strong correlation. Interestingly, each style vector encodes the style of a group of similarly pronounced phonemes, such as the group of ‘M’, ‘N’, and ‘NG’. This observation confirms that content-specific style is actually realized in Retriever features.

heading: 5.1.3 ZERO-SHOT VOICE CONVERSION TASK
text: Zero-shot voice conversion task converts the source speaker’s voice into any out-of-training-data speaker’s while preserving the linguistic content. We follow the setting of previous work to randomly select 1,000 test utterance pairs of 18 unseen speakers from the CMU Arctic databases (Kominek & Black, 2004). Each conversion is given five target utterances to extract the style. Both objective and subjective metrics are used to evaluate conversion similarity (SV accuracy, SMOS) and speech quality (MOSNet score, MOS). More details of the four metrics are in Appendix F.2.
Existing approaches fall into two categories: content-style disentanglement approaches (Qian et al., 2019; Yuan et al., 2021; Chou & Lee, 2019; Wu et al., 2020), and deep concatenative approaches (Lin et al., 2021b;a). Methods in the first category use a shared style embedding for all the content features while methods in the second category try to find the matching style for each content frame, resulting in high computational complexity. Our method provides a similar level of style granularity as the second class of approaches. But our bipartite graph representation is more compact and effective as it is capable of retrieving content-specific style with a light-weight link attention operation. Table 2 shows the comparison between our Retriever-based method and four SOTA methods. The superior performance demonstrates the power of the content-style bipartite graph representation. Audio samples are provided in the supplementary material.
5.1.4 ABLATION STUDY ON RETRIEVER
Based on the zero-shot voice conversion task, we carry out ablation studies on the VCTK + CMU Arctic dataset to better understand Retriever.
Style Encoder. Unlike most previous work, which are limited to learning a single speaker style vector, our method can learn an arbitrary number of style tokens.When the number of style tokens increases from 1 to 10 and 60, the SV accuracy increases from 81.3% to 94.3% and 99.4%, showing the benefits of fine-grained style modeling.
Content Encoder. We study how the capacity of the information bottleneck affects the separation of content and style. Compared to the proper bottleneck setting (G = 2, V = 100), speech quality significantly drops when the bottleneck is “too narrow” (G = 1, V = 10) and conversion similarity drops when the bottleneck is “too wide”(G = 8, V = 100), as shown in Table 3. This indicates content loss and style leakage when the bottleneck is too narrow and too wide, respectively.
Decoder. To illustrate the advantages of our link attention-based decoder, we replace it with the traditional AdaIN module for experiment. As the AdaIN module only takes a single vector as input, we flatten the 60 style tokens produced by Retriever into a very long vector. Results in Table 3 show that, although the amount of information provided to the decoder is the same, AdaIN-based decoder has a significant performance drop. This confirms the value of link attention.

heading: 5.2 IMAGE DOMAIN
text: The image domain is one of the first domains to carry out the research on content-style disentanglement. Conventionally, shape is treated as content and appearance is treated as style. Interestingly, our unified definition of content and style aligns with the intuitive understanding of images.
5.2.1 TRAINING RETRIEVER FOR IMAGES
To tokenize the image, we downsample it by 4 times using a stack of convolutions. For VQ module, we set G to 1, and V is dependent on the dataset. The convolution after VQ operation is of kernel size 3× 3. We detokenize the output tokens to images by convolutions and PixelShuffles (Shi et al., 2016). For the reconstruction loss Lrec, we apply a typical perceptual loss (Chen & Koltun, 2017). For the structural constraint LSC , we adopt a geometric concentration loss (Hung et al., 2019). We choose two commonly used datasets: Celeba-Wild (Liu et al., 2015) and DeepFashion (Liu et al., 2016). Images in Celeba-Wild are not aligned and each face has five landmark coordinates. The full-body images from DeepFashion are used. See Appendix E for more details and experiments.

heading: 5.2.2 UNSUPERVISED CO-PART SEGMENTATION
text: Unsupervised co-part segmentation aims to discover and segment semantic parts for an object. It indicates whether the content representations are semantically meaningful. Existing methods (Hung et al., 2019; Liu et al., 2021) rely heavily on hand-crafted dataset-specific priors, such as background assumption and transformations. We choose SCOPS (Hung et al., 2019) and Liu et al. (2021) as baselines and follow the same setting to use landmark regression on Celeba-Wild dataset as a proxy task for evaluation. Please refer to the Appendix E.2 for details.
The content code of Retriever is visualized in Figure 6 (a) for the Celeba-Wild dataset and in Figure 8 for the DeepFashion dataset. By setting a small V in Retriever, the encoded content only preserves the basic shape information as segmentation maps. Each content token is consistently aligned with a meaningful part among different images. When compared to the state-of-the-art
methods, as Table 4 shows, Retriever is one of the top performers on this task, even without any hand-crafted transformations or additional saliency maps.

heading: 5.2.3 UNSUPERVISED PART-LEVEL STYLE TRANSFER
text: Part-level style transfer requires disentanglement of shape and appearance at part-level, which is particularly challenging in an unsupervised manner. Previous works (Lorenz et al., 2019; Liu et al., 2021) propose to use one-one matching between shape and appearance, which is not flexible and leads to unrealistic results. Our content-style bipartite graph separates content and style at part-level and enables a more flexible content-specific style modeling. We can perform part-level style transfer by calculating the co-occurrence map between content and style tokens. See Appendix B for more details.
Both image-level and part-level transfer results are shown in Figure 6, 7, and 8. Retriever is capable of the explicit control of local appearance. Even without any supervision for the highly unaligned image pairs, our method can transfer the appearance to a target shape with high visual quality. See Appendix E.4 for more results.

heading: 6 CONCLUSION
text: In this paper, we have designed and implemented an unsupervised framework for learning separable and interpretable content-style features. At the core of the proposed Retriever framework are the two retrieval operations powered by the innovative use of multi-head attention. Cross-attention is used as the style encoder to retrieve style from the input structured data, and link attention is used as the decoder to retrieve content-specific style for data reconstruction. We have demonstrated that structural constraints can be integrated into our framework to improve the interpretability of the content. This interpretability is further propagated to the style through the innovative bipartite graph representation. As a result, the proposed Retriever enables a couple of fine-grained downstream tasks and achieves superior performance. As for the limitations, we have discovered in experiments that different tasks on different datasets need different model settings. We currently lack theoretical guidance for determining these settings, and we plan to work on it in the future.

heading: A BACKGROUD
text: Attention. Given n query vectors, each with dimension dq: Q ∈ Rn×dq , an attention function Attn(Q,K,V ) maps queries Q to outputs using m key-value pairs K ∈ Rm×dq ,V ∈ Rm×dv ,
Attn(Q,K,V ) = softmax( QKT√ dq )V ,
where dot product QKT ∈ Rn×m measures the similarity between queries and keys. softmax(·) normalizes this similarity matrix to serve as the weight of combining the m elements in V .
Multi-Head Attention. Vaswani et al. (2017) propose multi-head attention, denoted as MHA(·, ·, ·). It projects Q,K,V into h different groups of query, key, and value vectors, and applies attention function on each group. The output is a linear transformation of the concatenation of all attention outputs:
Oi = Attn(QW Q i ,KW K i ,V W V i ),
MHA(Q,K,V ) = concat(O1,O2, ...,Oh)W O,
where WQi ,W K i ∈ Rdq× dq h , and W Vi ∈ Rdv× dv h are the projection matrix for query, key, and value in the i-th head, WO ∈ Rdv×dq is the linear transformation that fuses all the heads’ outputs. Self-Attention. Given an input array X ∈ Rn×dx , a self-attention SA(·) or multi-head self-attention MHSA(·) operation takes X as query, key, and value:
SA(X) = Attn(X,X,X)
MHSA(X) = MHA(X,X,X)
Cross-Attention. Given input arrays X ∈ Rn×dx ,Y ∈ Rm×dy , a cross-attention CA(·, ·) or multihead cross-attention MHCA(·, ·) operation takes X as query, and takes Y as key and value:
CA(X,Y ) = Attn(X,Y ,Y )
MHCA(X,Y ) = MHA(X,Y ,Y )
Property: (multi-head) cross-attention operation is P.I. to Y , that is,
∀π ∈ Pm,MHCA(X, π(Y )) = MHCA(X,Y ),
where Pm is the set of all permutations of indices {1, ...,m} Permutation Invariant Property of Style Encoder For the i-th style encoder block, we assume its input Zi−1 is P.I. to F . The output of the multi-head cross-attention layer is Ẑi = MHCA(Zi−1,F ). According to the property of multi-head cross-attention, we have Ẑi is permutation invariant to F . The following residue connection combines Ẑi with Zi−1, both of which are P.I. to F . Thus the combination is P.I. to F . The following token mixing layer and feed-forward layer don’t take F as input. Because their inputs are P.I. to F , their outputs are also P.I. to F . To summarize, if Zi−1 is P.I. to F , then Zi is also P.I. to F . It is obvious that Z0 is P.I. to F since Z0 is fixed during the forward path. Therefore, the output of the whole style encoder is P.I. to F .
Vector Quantization. For efficiency, we use product quantization (Baevski et al., 2020a;b). The quantized feature ci is the concatenation ofG embeddings e1, e2, ..., eG ∈ Rdc/G, which are looked up from the corresponding G codebooks E1,E2, ...,EG ∈ RV×dc/G, where V is the number of entries in each codebook. To make VQ operation differentiable, Gumbel-softmax (Jang et al., 2017) is adopted: Firstly, each input token is mapped to logits l ∈ RG×V . Then, eg is calculated by a weighted sum of codewords in codebook Eg , and the weight for adding the v-th code in group g is given by:
wg,v = exp((lg,v + ng,v)/τ)∑V k=1 exp((lg,k + ng,k)/τ) , (2)
where τ is temperature, ng,v = − log(− log(ug,v)), and ug,v ∼ U(0, 1) is independently sampled for all subscripts. We use the straight-through estimator (Jang et al., 2017), treating the forward and backward path differently. During the forward path, the one-hot version of the above weight
one hot(argmaxk wg,k) is used, and in the backward path, the gradient of the original wg,k is used. To encourage all the VQ codes to be equally used, we use a batch-level VQ perplexity loss:
LV Q = 1
GV G∑ g=1 exp(−H(pg)) = 1 GV G∑ g=1 exp( V∑ v=1 pg,vlogpg,v)), (3)
where pg indicates the probability calculated by softmax(lg) averaged across all the tokens in batch, H(·) calculates the entropy of a discrete distribution.
B VISUALIZATION OF LEARNED CONTENT-STYLE BIPARTITE GRAPH
Retriever models the content-style bipartite graph by introducing the novel link attention module. The graph’s edges are represented by the link attention map. To understand what content is linked to which style, we gather the statistics of the link attention map as the content-style cooccurrence map. To show content information in a 2-d map, the content is categorized by either the ground-truth labels, or the discovered parts or units. Specifically, the co-occurrence statistics of content category c and style token s, denoted as mc,s, is calculated as follows:
mc,s = 1 |D| ∑ x∈D A(x, s)I(l(x) = c),
whereD indicates the set composed of all the content tokens in the target dataset. A(x, s) is the link attention amplitude between content token x and style token s. I(·) is the characteristic function. It outputs 1 if its input is true and outputs 0 if its input is false. l(x) gives the content category of token x.
For each style token, we consider the content category that shows the strongest co-occurrence with it to be the major content category that it serves. For the convenience of finding the pattern inside the co-occurrence map, we normalize on both content category axis and style token axis, then sort the style tokens so that their major content category index is in the ascending order.
In audio domain, we categorize content tokens using ground-truth phoneme labels and calculate the co-occurrence map using the first link attention. The content-style co-occurrence map is shown in Figure 9.
In image domain, we categorize content tokens using the discovered parts and calculate the cooccurrence map using the third link attention. The part indices together with content-style cooccurrence map are shown in Figure 10.

heading: C STYLE SPACE VISUALIZATION
text: In order to provide an intuitive impression about what the styles decomposed by our framework are, we present visualizations of some style spaces produced by t-SNE (Van der Maaten & Hinton, 2008). We see the clustered patterns in many style spaces which correspond to meaningful attributes like person gender and clothes color.
C.1 SPEECH DOMAIN
We visualize the style vectors extracted from 500 randomly-selected utterances of ten unseen speakers from Librispeech test-clean split. In Figure 11, we show the latent space of four typical style vectors, which correspond to the style of nasal consonant (NG), voiceless consonant (T), voiced consonant (Z), and vowel (IH). Different speakers are labeled in different colors. We see that different speakers are clustered into different clusters.
C.2 IMAGE DOMAIN
We examine the style vector spaces on Deepfashion dataset, and find that some style vector spaces are separately or collectively correlated with the human-defined attributes. For example, we find that the style vector #43 is highly correlated with the color of the clothes. Figure 12 shows the t-SNE visualization of this vector space, as well as some images in each cluster. It is clear that images are clustered solely based on the color of the clothes despite of various model identifications, standing poses, and styles of the dress.
As this dataset also has the gender attribute labelled, we further look for the ‘gender vector’ in the decomposed style vector spaces. Interestingly, we find 14 vector spaces that are highly correlated to the gender attribute. We concatenate these 14 vectors and show the t-SNE visualization in Figure 13.
We can see a clear boundary between male and female models despite of the their color of clothes or standing poses.

heading: D STRUCTURAL CONSTRAINT
text: D.1 IMAGE DOMAIN
To make the content representation interpretable, we can apply additional structural constraint. Inspired by the observation that pixels belonging to the same object part are usually spatially concentrated and form a connected region, we follow Hung et al. (2019) to use a geometrical concentration loss as the structural constraint to shape the content representation to part segments. In practice, we denote the pixels assigned to the first entry in the VQ codebook as background pixels and the other pixels as the foreground pixels.
Given the logits L ∈ RV×H×W that assign each pixel to the parts (entries in VQ codebook), for foreground pixels:
Lfg = 1
V − 1 V∑ v=2 ∑ h,w ∥∥∥∥[hw ] − [ cvh cvw ]∥∥∥∥2 2 · Lv,h,w/zv, (4)
where ckh and c k w are the coordinates of the v-th part center along axis h, w and zk is a normalization term. ckh, c k w and zk can be calculated as follows:
cvh = ∑ h,w h · Lv,h,w/zv, cvw = ∑ h,w w · Lv,h,w/zv, (5)
zv = ∑ h,w Lv,h,w. (6)
For the background pixels, different from previous methods using background loss (Liu et al., 2021) or saliency map to bound the region of interested object, we do not add any constraint. Thus LSC = Lfg
D.2 AUDIO DOMAIN
To discover phoneme-like units in speech, the discrete representation should not switch too frequently along the time axis. Therefore, we penalize the code switch based on the calculation of neighborhood cross-entropy.
LCE = 1
N − 1 N−1∑ i=1 CE(pi, zi+1) + CE(pi+1, zi), (7)
where N is the length of the input sequence, CE(·) calculates the cross-entropy between discrete probability and given label, pi denotes the predicted discrete probability before Gumbel-softmax, and zi is the highest-probability code on position i.
However, LCE also penalizes the true phoneme switch, where the input acoustic feature really changes dramatically. To solve this problem, we use a truncated version of LCE as our structural constraint:
LSC = γtanh( LCE γ ), (8)
where γ is a hyper-parameter indicating the truncation threshold. For the true phoneme switch, high LCE is expected, thus the loss will be truncated, while within the same phoneme, low LCE is expected, and LSC behaves like the original CE loss. As such, the truncated loss is more suitable for discovering phoneme-like units. In our experiment, we set γ = ln(V ) = ln(100).
E IMAGE: ADDITIONAL DETAILS
E.1 IMPLEMENTATION DETAILS
We first show the tokenization module in Table 5 and the detokenization module in Table 6. For the layer numbers, we set Le, Ls, Ld to 6, 3, 6, respectively. The hyper-parameter settings of each styleencoder layer and decoder layer are shown in Table 7 and Table 8, where dFFN indicates the hidden
dimension of the feed-forward network, and nhead is the number of heads used in the multi-head attention modules. Specifically, for the decoder, we replace the FFN with Mix-FFN to improve the image quality. The Mix-FFN can be formulated as:
xout = MLP(GELU(Conv3×3(MLP(xin)))),
where xin is the input feature. The Gumbel-Softmax temperature τ in VQ is annealed from 2 to a minimum of 0.01 by a factor of 0.9996 at every update.
For the Celeba-Wild dataset, we resize the input to 128 × 128. For DeepFashion dataset, we resize the input to 192×96. For the number of style tokens, we use 512. We summarize the training setting in Table 9. Our model is implemented with Pytorch and trained on 4 Nvidia V100 GPUs.
Table 5: Tokenization module for images.
Conv 3× 3× 3× 24, stride = 1 BatchNorm ReLu Conv 3× 3× 24× 48, stride = 2 BatchNorm ReLu Conv 3× 3× 48× 96, stride = 1 BatchNorm ReLu Conv 3× 3× 96× 192, stride = 2 BatchNorm ReLu Conv 3× 3× 192× 192, stride = 1
Table 6: Detokenization module for images.
PixelShuffle(2) Conv 3× 3× 48× 48, stride = 1 ReLu PixelShuffle(2) Conv 3× 3× 12× 12, stride = 1 ReLu Conv 1× 1× 12× 3, stride = 1
E.2 DETAILS ABOUT EVALUATION ON CO-PART SEGMENTATION
For the evaluation on co-part segmentation, we follow the setting in Liu et al. (2021) and Hung et al. (2019). The proxy task for Celeba-Wild is landmark regression. We first convert part segmentations into landmarks by taking part centers as in Eq. 5. Then we learn a linear regressor to map the converted landmarks to ground-truth landmarks and evaluate the regression error on test data.
E.3 ABLATION STUDY
Do the style encoder and decoder impact the co-part segmentation? In this work, we find VQ module is suitable for unsupervised co-part segmentation, as it is a cluster center among the whole dataset. Here we evaluate that if the style branch further helps the co-part segmentation. As shown in Table 10, with a style branch, the discovered part segmentation is more consistent. Moreover, we compare our link attention decoder with AdaIN operation and find that our design is better for co-part segmentation task. This further verified that our content-style bipartite graph is a powerful representation with high interpretability.
On using different structural constraint weight. We further carry out experiments with different loss weight for structural constraint. Without the structural constraint, the parts learned are scattered all over an image. As the weight for structural constraint getting larger, the part segmentation becomes tighter. With too large weight, the regions lose flexibility and fail to cover the parts.
On using different VQ diversity loss weight. We also test our model’s sensitivity to λvq . As long as λvq is not set to zero or too large (e.g., λvq = 1), the model has a fairly stable landmark regression error even when λvq is adjusted from 0.01 to 0.3.
On using different tokenization module. Different tokenization modules bring additional bias. Besides a stack of convolution layers to tokenize the input image, we also experiment with the PatchMerge operation in ViT (Dosovitskiy et al., 2021), which is a 4 × 4 convolution layer with stride 4 in our case. As shown in Table 10, using convolution layers as the tokenization module can help the preprocessing transformer blocks inside Retriever work better, which is also observed by a concurrent work (Xiao et al., 2021).
On using different downsampling rate. In this work, we mainly use 4× downsampling rate. For the input 128 × 128 image, we first tokenize it to 32 × 32. Here we experiment with additional different downsampling rates: 8× and 16×. In addition to the convolution layer for 4× setting, we add additional convolution layers for further downsampling. We do not try 2× due to the limited computational resource. As shown in Table 10, for the selected downsampling rate, larger one results in worse performance. We also provide visual comparison in Figure 16.
E.4 MORE QUALITATIVE RESULTS
Co-part segmentation. We first provide more visualization on Celeba-Wild in Figure 17. Besides, we also apply our method on CUB dataset (Wah et al., 2011). CUB is a dataset with 11, 788 photos of 200 bird species in total. This dataset is challenging due to the pose of the birds is of large diversity. We follow (Liu et al., 2021) select the photos from the first 3 categories, and get about 200 photos. The results are shown in Figure 18.
Part-level style transfer. Besides the results in the main paper, we also apply our method on a high-resolution dataset: CelebA-HQ. We resize the images to 256 × 256. We show the part-level style transfer results in Figure 19 for mouth, nose, and eye.
Zero-shot image style transfer. We find our model capable of generalizing into unseen image datasets. In Figure 20, we use our model trained on CelebA-Wild to transfer artistic style from an artistic image. The transfer result keeps the pose and shape of the content images (the first row), while adopting the tone and appearance of the style image (the first column on the left).
Visual comparison. We find our method enables more natural style transfer compared with the previous work. Here we show side-by-side visual comparison on Deepfashion dataset in Figure 21 and 22. In image-level style transfer, the results of Lorenz et al. (2019) suffer from significant deformation artifact. In contrast, such artifact is not observed in our results. In part-level style transfer, we show the results of head transfer. Again, our results are more natural and contain more image details.

heading: F AUDIO: ADDITIONAL DETAILS
text: F.1 IMPLEMENTATION DETAILS
For the tokenization module, we follow Lin et al. (2021a), using s3prl toolkit1 to extract the CPC feature, and use a depth-wise convolutional layer with kernel size 15 after the CPC feature. The depth-wise convolutional layer is trainable, while the CPC model is fixed during training. For the detokenization module, we use publicly available implementation of Parallel WaveGAN2 pretrained on LibriTTS. Le, Ls, Ld are set to 0, 3, 4, respectively. The dimension of content representation is dc = 512. Following Qian et al. (2020), we add auxiliary quantized normalized f0 feature into content for better voice quality. The Gumbel-Softmax temperature τ is annealed from 2 to a minimum of 0.01 by a factor of 0.9996 at every update. The hyper-parameter settings of each style-encoder layer and decoder layer are shown in Table 11 and Table 12, where dFFN indicates the hidden dimension of the feed-forward network, and nhead is the number of heads used in the multi-head attention modules. The decoder output is converted to 80-d log-Mel spectrum with an MLP whose hidden dimension is 4096. The training hyper-parameter setting is listed in Table 13. Each training sample is a random 4-second audio clip sampled from the training dataset. For utterances that are shorter than 4s, we pad zero at the end of the utterance. The code is implemented with Pytorch, and training takes 5 hours on 4 Nvidia V100 GPUs.
F.2 EVALUATION METRICS OF ZERO-SHOT VC TASK
For objective metrics, conversion similarity is measured by Resemblyzer speaker verification system 3 by calculating speaker verification accuracy (SV Accu) between converted speech and corresponding target utterance, as done in the previous work (Lin et al., 2021b). The conversion is considered successful if the cosine similarity between Resemblyzer speaker embedding of converted speech and target utterance exceeds a pre-defined threshold. The threshold is decided based on equal error rate (EER) of the SV system over the whole testing dataset. The SV accuracy is the percentage of successful conversions. The objective speech quality metric is estimated by MOSNet (Lo et al., 2019). It takes converted speech as input and outputs a number ranging from 1 to 5 as the measurement of speech naturalness. Both metrics are higher the better.
1https://github.com/s3prl/s3prl 2https://github.com/kan-bayashi/ParallelWaveGAN 3https://github.com/resemble-ai/Resemblyzer
For subjective metrics, we conduct two tests to calculate the mean opinion score of conversion similarity (denoted as SMOS), and speech quality (denoted as MOS). For conversion similarity, each subject is asked to listen to the target utterance and converted utterance, and judge how confident they are spoken by the same speaker. For speech quality, each subject is asked to listen to utterances that are randomly chosen from converted speech and real speech, and determine how natural they sound. Both test results are scored from 1 to 5, the higher, the better. We randomly sample 40 utterances from the test set for both tests. Each utterance is evaluated by at least 5 subjects. The score is then averaged and reported with 95% confidence intervals.
F.3 PHONEME INFORMATION PROBING: TRAINING AND TESTING
We conduct phoneme information probing to test how much phoneme information is encoded in the learned discrete content tokens, so as to demonstrate their interpretability. We experiment with two settings, one considers the contextual information (denoted as “Frame w/ context”), and the other considers only single-frame information (denoted as “Frame”). For these two settings, a 1d convolutional layer with kernel size 17 or a linear layer is used as the probing network respectively, to predict ground-truth phoneme label. For the input of the probing network, each frame is represented by the one-hot vector of the corresponding code. For the experiment involving two groups of code, the one-hot vector of both groups are concatenated on the channel axis. For the k-means clustering of CPC feature, the cluster number is set to 100, which is the same as the number of codebook entries per VQ group. We train the probing network on LibriSpeech train-clean-100 split and test it on LibriSpeech test-clean split. Adam optimizer is used with β1 = 0.9, β2 = 0.999. Learning rate and batch size are set to 0.00005 and 30, respectively. Each training and testing sample is a random 2-second segment from the dataset. We drop the utterance that is shorter than 2 seconds. In Table 1, we report the test accuracy after the convergence of the training.
F.4 DETAILS IN SYSTEM COMPARISON
We choose four SOTA systems for comparison. Two content-style disentanglement approaches are AutoVC (Qian et al., 2019)4, and AdaIN-VC (Chou & Lee, 2019)5. Two deep concatenative approaches are FragmentVC (Lin et al., 2021b)6, and S2VC (Lin et al., 2021a)7. All these methods have officially released their code, together with the pretrained models also on VCTK dataset. We adopt these models and test them on our randomly sampled 1000 conversion pairs in CMU Arctic dataset.
F.5 INFERENCE SCALABILITY ON ZERO-SHOT VC TASK
Table 14 shows the inference scalability of our method on VCTK + CMU Arctic dataset. “# target” indicates the number of available target utterances for inference. We see that Retriever performs quite well in the case that only one target utterance is given. As the available target utterance increases, the conver-
sion similarity keeps increasing up to 99.6%, and it has no negative effect on speech quality, indicating the extracted style becomes more accurate when seeing more target samples.
F.6 PERFORMANCE ON LIBRISPEECH DATASET
LibriSpeech dataset (Panayotov et al., 2015) is more diverse than VCTK + CMU Arctic in the aspect of emotion, vocabulary, and speaker number, thus closer to real-world application. In this experiment, the whole train-clean-100 split containing 251 speakers is used for training, and testclean split containing 40 speakers is used for testing. When testing, the conversion source-target pairs are built as follows: for each utterance in the test set, we treat it as the source utterance, and
4https://github.com/auspicious3000/autovc 5https://github.com/jjery2243542/adaptive_voice_conversion 6https://github.com/yistLin/FragmentVC 7https://github.com/howard1337/S2VC
assign it one target utterance that is randomly sampled from the test set. In this way, each utterance in the dataset serves as the source for one time.
Objective similarity metric is measured for using different style token numbers, as shown in Figure 23. The conversion similarity increases when using more style tokens, and the performance saturates at nearly 100%. To be specific, we achieve 98.4% SV accuracy and 3.13 MOSNet score when using 60 style tokens, demonstrating that our method can be generalized into more complicated scenarios and potentially can be used in the real-world application.
F.7 LOSS WEIGHT ABLATION STUDY
To test the effectiveness of structural constraint and VQ diversity loss, we do ablation study on the corresponding loss weights, λV Q and λSC . The SV accuracy and MOSNet scores achieved at different parameter settings are shown in Table 15. We observe that the absence of either loss term leads to noticeable speech quality (MOSNet score) drop. We further test the model’s sensitivity to these two loss weights and we are able to empirically conclude that our framework is not sensitive to the selection of these two hyper-parameters. When λSC increases from 0.002 to 0.1 (by 50 times), SV accuracy only fluctuates by ±0.45% and MOSNet score only fluctuates by ±0.01. When λV Q increases from 0.01 to 2.0 (by 200 times), SV accuracy only fluctuates by ±0.05% and MOSNet score only fluctuates by ±0.02. In a nutshell, both structural constraint and VQ diversity loss are necessary, and our model is not sensitive to the selection of the corresponding loss weights in a reasonably large range.

Review summaries:
The paper proposes a novel idea of disentangling the content and style of structured data by treating style as permutation invariant information. It adopts VQ network for content encoding, and Cross-Attention for Style and Linking Attention at decoder. It is shown to be domain agonistic - worked well in image and audio domain.

This paper proposes a framework for learning disentangled representations of content and style in an unsupervised way. The paper presents a slightly uncommon definition of content and style, which serves as a foundation for the proposed methodology and experiments. Experiments are conducted on speech and image datasets. 

This paper proposes an unsupervised framework to learn content and style representations of structured data, such as texts, images and speeches. The task is interesting to me and decomposition representation learning is important. In addition, a bipartite graph is proposed to model the link between content tokens and style tokens, making the model more interpretable. The proposed model performs relatively well on both speech and image domains.