Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph
Authors: Dacheng Yin
Xuanchi Ren
Chong Luo
Yuwang Wang
Zhiwei Xiong
Wenjun Zeng
Venue: ICLR 2022 Poster (2022)
Keywords: Content-style decomposed representation
Zero-shot voice conversion
Style transfer
Transformer
Unsupervised learning

Abstract: This paper addresses the unsupervised learning of content-style decomposed representation. We first give a definition of style and then model the content-style representation as a token-level bipartite graph. An unsupervised framework, named Retriever, is proposed to learn such representations. First, a cross-attention module is employed to retrieve permutation invariant (P.I.) information, defined as style, from the input data. Second, a vector quantization (VQ) module is used, together with man-induced constraints, to produce interpretable content tokens. Last, an innovative link attention module serves as the decoder to reconstruct data from the decomposed content and style, with the help of the linking keys. Being modal-agnostic, the proposed Retriever is evaluated in both speech and image domains. The state-of-the-art zero-shot voice conversion performance confirms the disentangling ability of our framework. Top performance is also achieved in the part discovery task for images, verifying the interpretability of our representation. In addition, the vivid part-based style transfer quality demonstrates the potential of Retriever to support various fascinating generative tasks. Project page at https://ydcustc.github.io/retriever-demo/.

Published as a conference paper at ICLR 2022
RETRIEVER: LEARNING CONTENT-STYLE REPRESEN- TATION AS A TOKEN-LEVEL BIPARTITE GRAPH
Dacheng Yin 'Xuanchi Ren Ren2,,Chng Luo Yuwang Wang" Xiong' ,Wenjun Zeng* University of Science and Technology of China, ²HKUST, "Microsoft Research Asia, "EIT [ydeemail., zwxionge Justc..t...... edu. en, *xrenaa8connect. ust.hk hk, 3 {cluo, yuwann)emicrosoffe com,
wennjunzenggeias.acc
ABSTRACT
This paper addresses the unsupervised learning of content-style decomposed rep- resentation. We first give definition of style and then model the content-style rep- resentation as token-level bipartite graph. An unsupervised framework, named Retriever, is proposed to learn such representations. First, cross-attention module is employed to retrieve permutation invariant (P...) information, defined style, from the input data. Second, vector quantization (VQ) module is used, together with man-induced constraints, to produce interpretable content tokens. Last, innovative link attention module serves the decoder reconstruct data from the decomposed content and style, with the help of the linking keys. Being modal-agnostic, the proposed Retriever is evaluated in both speech and image domains. The tftt--ff-the-ar zero-shot voice conversion performance confirms the disentangling ability of our framework. Top performance is also achieved the part discovery task for images, verifying the interpretability of our represen- tation. addition, the vivid part-based style transfer quality demonstrates the potential of Retriever to support various fascinating generative tasks. Project page at
github. wtt.:oi//eiiiii. demo/
INTRODUCTION
Human perceptual systems routinely separate content and style to better understand their observa- tions (Tenenbaum & Freeman 2000). In artificial intelligence, content and style decomposed representation is also very much desired. However, we notice that existing work does not have unified definition for content and style. Some definitions are dataset-dependent (Chou Lee 2019 Ren et al. 2021), while some others have to be defined on certain modality (Lorenz et al. 2019: Wu al. 2019). We wonder, since content-style separation is helpful our entire perception system, why is there not unified definition that applies to all perception data?
In order to answer this question, we must first study the characteristics data. The data of inter- est, including text, speech, image, and video, are structured. They can be divided into standardized tokens, either naturally as words language and speech or intentionally as patches in images. No- tably, the order of these tokens matters. Disrupting the order of words can make speech express completely different meaning. Reversing the order of frames in a video can make a stand-up action become sit-down action. But there is also some information that is not affected by the order of tokens. For example, scrambling the words in a speech does not change the speaker's voice, and a frame-shufiled video does not change how the person in the video looks. We notice that the content we intuitively think of is affected by the order of tokens, while the style is usually not. Therefore, we could generally define style as token-level permutation invariant (P...) information and define content as the rest of the information in structured data.
However, merely dividing data into two parts is not enough. As Bengio et al. (2013) pointed out, we are to take the notion of disentangling seriously, we require richer interaction of features than that offered by simple linear combinations. An intuitive example that we could never generate
Equal contribution. Work done during internships at Microsoft Research Asia.
1 Joint corresponding author.

---

Published as a conference paper at ICLR 2022
array content tokens
4
2 be,
3
or
not
10
12
13
14
16
17
18
That
the question
22
set style tokens
Figure 1: Illustration of the proposed modal modal-agnostic bipartite-gran representation of content and style. The content an array of position-sensitvee sensitive tokens and the style is set of P.I. tokens.
a colored pattern by linearly combining a generic color feature with a gray-scale stimulus pattern. Inspired by this, we propose to model content and style by token-level bipartite graph, Figure] illustrates. This representation includes an array of content tokens, set of style tokens, and a set of links modeling the interaction between them. Such representation allows for fine-grained access and manipulation of features, and enables exciting downstream tasks such as part-level style transfer.
In this paper, we design modal-agnostic framework, named Retriever, for learning bipartite graph representation of content and style. Retriever adopts the autoencoder architecture and addresses two main challenges in it. One is how to decompose the defined content and style in an unsupervised setting. The other is how to compose these two separated factors to reconstruct data.
To tackle the first challenge, we employ a cross-attention module that takes the datasct-shared pro- totype as query to retrieve the style tokens from input data (Carion et al. (020). A cross-attention operation only allows the P.I. information to pass (Lee al.,)201)) which exactly what we want for style. On the other content path, we employ vector quantization (VQ) module (van den Oord jet al., 2017) as the information bottleneck. In addition, we enforce man-induced constraints to make the content tokens interpretable. To tackle the second challenge, we innovate the link attention mod- ule for the reconstruction from the bipartite graph. Specifically, the content and style serve as the query and value, respectively. Links between the content and style are learnt and stored in the link- ing keys. Link attention allows us to retrieve style by content query. As such, the interpretability is propagated from content style, and the entire representation is friendly fine-grained editing.
We evaluate Retriever in both speech and image domains. In the speech domain, we achieve state-of-the-art (SOTA) performance in zero-shot voice conversion, demonstrating complete and precise decomposition of content and style. In the image domain, we achieve competitive results part discovery task, which demonstrates the interpretability of the decomposed content. More excitingly, we try part-level style transfer. which cannot be offered by most of the existing content- style disentanglement approaches. Vivid and interpretable results are achieved.
To summarize, our main contributions are three-folds: i) We provide an intuitive and modal- agnostic definition of content and style for structured data. We are the first to model content and style with a token-level bipartite graph. ii) We propose an unsupervised framework, named Retriever, for learning the proposed content-style representation. A novel link attention module is designed for data reconstruction from content-style bipartite graph. iii) We demonstrate the power of Retriever in challenging downstream tasks in both speech and image domains.
RELATED WORK
Content-style decomposed representation can be approached in supervised or unsupervised settings. When style labels, such the speaker labels of speeches (Kameoka et al., 22088; Qian et al. 2019
Yuan et al. 2021) and the identity labels of face images (Mathicu et al. 2016:) Szabó et al. 2018
Tha et al. 2018 Bouchacourt al. 2018: Gabbay & Hoshen (020), available, latent variables can be divided into content and style based on group supervision.
Recently, there has been increased interest in unsupervised learning of content and style. Since there no explicit supervision signal, the basic problem one must first solve is the definition of content and style. We discover that all existing definitions are either domain-specific task-specific. For
2

---

Published as a conference paper at ICLR 2022
Style
Content
information
fax
()
fan
Tex
gtts
e(>)
glr)
fax
fu
(a) No bottleneck
(b) Wide bottleneck
(c) Perfect bottleneck
(d) Narrow battleneck
Figure An illustration of our content-style separation mechanism.
example, in speech domain, Chou & Lee (2019) assume that style is the global statistical informa- tion and content is what is left after instance normalization (IN). Ebbers et al. (2021) suggest that style captures long-term stability and content captures short-term variations. In image domain, the definition is even more diverse. Lorenz et al. (2019) try to discover the invariants under spatial and appearance transformations and treat them as style and content, respectively. [Wu al., (2019) define content as 2D landmarks and style as the rest of information. Ren et al. (2021) define content as the most important factor across the whole dataset for image reconstruction, which is rather abstract. In this work, we attempt to find a general and modal-agnostic definition of content and style.
Style transfer is partially related to our work, as it concerns the combination of content and style to reconstruct data. AdaIN (Huang & Belongie, 2017, Chou & Lee, 2019) goes beyond the linear combination of content and style (Tenenbaum & Freeman 2000) and proposes to inject style into content by aligning the mean and variance of the content features with those of the style features. However, style is not separated from content in this line of research. Liu et al. (2021) touch upon the part-based style transfer task we do. They model the relationship between content and style by a one-one mapping. They follow the common definition of content and style in the image domain as shape and appearance, and try disentangle them with hand-crafted data augmentation methods.
Besides, the term "style" is often seen in image generative models, such as StyleGAN (Karras et al., 2019). However, the style mentioned in this type of work is conceptually different from the style in our work. In StyleGAN, there is no concept of content, and style is the whole latent variable con- taining all the information including the appearance and shape of an image. Following StyleGAN, Hudson & Zitnick (202T) employ a bipartite structure to enable long-range interactions across the image, which iteratively propagates information from a set of latent variables to the evolving visual features. Recently, researchers have become interested in disentangling content and style from the latent variables of StyleGAN (Alharbi & Wonka 2020 Kwon & Ye, (021). However, they only work for well-aligned images and are hard to be applied other modalities.
3 CONTENT-STYLE REPRESENTATION FOR STRUCTURED DATA
In this section, we provide definitions of content and style for structured data, introduce the frame- work for content-style decomposition, and propose the token-level bipartite graph representation.
3.1 DEFINITION OF CONTENT AND STYLE
The data of interest is structured data that can be tokenized, denoted by X [21 22 T.,) We think of text, speech, and image, among many others, as structured data. Each token æ; can be word in text, phoneme in speech, non-triviae instances (examples of trivial instances are silent speech a blank image) are not able keep their full information when the order of tokens is not given. Inspired by this intuition, we define style of X as the information that is not affected by the permutation of tokens, or permutation invariant (P...) information. Content the rest of information X.
patch in image. These data are structured because
3.2 CONTENT-STYLE SEPARATION
The information in piece of structured data is carried either in the content in the style. By definition, style can be extracted by a P.I. function fr.r.(). which satisfies f(z(X)) = f(X), where TT-) represents permutation of tokens. To achieve content-style decomposition, naturally
3

---

Published as a conference paper at ICLR 2022
Content Encoder
Pre-Processig
vQ (Non-P1.)
HMM
Decoder
Oudput
NKNI
Ieput
p
Lnk
MUM
xf.
Style Tokless Content Styte Siportite Groph)
Prototypes
Style Encoder
Figure 3: Overview of the proposed Ret riever framework. The name is dubbed because of the dual-retrieval operations: the cross-attention module retrieves style for content-style separation, and the link attention module retrieves content-specific style for data reconstruction.
adopt an autoencoder architecture, as shown Figure 2 The bottom path is the style path which implements (.). We shall find powerful P.I. function which will let all the P.I. information pass. The top path is responsible for extracting the content, but the challenge is that there does not exist function which only lets pass the non-P.L. information. Therefore, we employ in the content path function, which will let all information pass, including the P.I. information. Obviously, if we do not pose any constraint information will be leaked to the content path. To squeeze the style information out of the content path, an information bottleneck fB.N.() (-) is required. A perfect bottleneck, as shown in Figure (c), can avoid the style leak while achieving perfect reconstruction. But an imperfect bottleneck, being too wide or too narrow, will cause style leak or content loss, as shown in Figure (b) and Figure (d), respectively.
a permutation-variian
the content path, shown in Figure (a), the style
3.3 TOKEN-LEVEL BIPARTITE GRAPH REPRESENTATION OF CONTENT AND STYLE
After dividing the structured input X into content feature C and style feature S, we continue to explore how the relationship between C and S should be modeled. Ideally, do not want all-to-one mapping, in which the style is applied to all the content as whole. Nor do we want one-to-one mapping where each content token is associated with fixed style. In order to provide flexible way of interaction, we propose a novel token-level bipartite graph modeling of C' and S.
The token-level representations of C and S are respectively. Note that there is a one-one correspondence between æ; and c,, so the order of the structured data preserved in the content C. While the order is preserved, the semantic meaning of each is not fixed. This suggests that the bipartite graph between C' and S cannot be static. In order model a dynamic bipartite graph, we introduce set of learnable linking keys K (ki, k2.... km), J. The linking keys and style tokens form set of key-value pairs {kk, 8;) linking key design allows for soft and learnable combination of content and style. The connection weight between content token C; and style token sj is calculated by Wi, ro(c,, ky), where ro is a learnable linking function parameterized with 6. For content token C, its content-specifi style feature now be calculated by su(c.) Σラ= Ui where ú; normalized weights.
[c1, C2,..., Cnl and 5 {s1 $2..... 8m }.
)m {:)) Our
THE RETRIEVER FRAMEWORK
4.1 OVERVIEW
This section presents the design and implementation of the Retriever framework, which realizes the bipartite graph representation of content and style as described in Section 3 Ret ever, as shown in Figure 3 is an autoencoder comprising of four major blocks, two of which implement the most important style retrieval functions. This why the name Retriever coined.
Retri is modal-agnostic.
The input of the framework is array of input tokens, denoted by X = [xi æ2,... xx, E Rnxd obtained by modal-speeific tokenization operations from raw data. The first major component is a pre- pre-processing module extract the feature F B from X. We implement it as a stack (L,) of non-P... transformer encoder blocks (Vaswani et al. 2017) to keep all the important information. Then F is decomposed by the content encoder [B.M..))
4

---

Published as a conference paper at ICLR 2022
Tolten Maxer
Convolution
Soll Attortiin
Link Pnneonttion
FFN
Norm
Foom
mx0.
rwd
[a] Cross-atten retrieve style from input
(b) to retrieve coptent-specific style for reconstruction
Figure 4: Implementation of (a) the style encoder and (b) the decoder in Retrieven
and the style encoder f-...()). fB.N is implemented by vector quantization to extract content C' (c1, C22 ...: Ca). Structural constraint be imposed C to improve interpretability. A cross-attention module is employed as (X) to retrieve style S = {s1, 82,..., 8m At last, the decoder, implemented by the novel link attention module, reconstructs X from the bipartite graph.
4.2 CONTENT ENCODER
Content encoder should serve two purposes. First, should implement an information bottleneck to squeeze out the style information. Second, it should provide interpretability such that meaningful feature editing can be enabled.
Information bottleneck.
Vector quantization (VQ) is good candidate of information bottleneck (Wu et al. 2020). though many other choices also easily fit into our framework. VQ maps data from continuous space into a set of discrete representations with restricted information capacity, and has shown strong generalization ability among data formats, such as image (van den Oord et 2017
Razavi et al. 2019, Esser et al. (021), audio (Baevski et al. 2020a b) and video Yaan et al. 2021).
We use product quantization (Baevski et al. 2020a b), which represents the input F with a concate- nation of G groups of codes, and each group of codes is from independent codebook of V entries. To encourage all the VQ codes to be equally used, we use a batch-level VQ perplexity loss, denoted Lvq. Please refer Appendix A for more details. When the bottleneck capacity is appropriate, the content can be extracted C VQ(F), where C e gaxd.
Interpretability. Priori knowledge can be imposed on the content path of Retriever to increase the interpretability of feature representations. In this work, we introduce structural constraint Lsc to demonstrate this capability. Other prior knowledge, including modal-specific ones, can be added to the framework long they can be translated into differentiable loss functions.
The structural constraint we have implemented in Retriever is a quite general one, as it reflects the locality bias widely exists in natural signals. In image domain, force the spatially adjacent tokens to share the same VQ code, so that single VQ code may represent a meaningful object part. In speech domain, we discourage the VQ code from changing too frequently along the temporal axis. As such, adjacent speech tokens can share the same code that may represent a phoneme. The visualization in the next section will demonstrate the interpretability of Retriever features.
4.3 STYLE ENCODER WITH
We have defined style as the P.I. information that can be extracted by a P.I. function from the struc- tured data.
which is widely used in set prediction task (Carion et al. 2020) and set data processing such 3D point cloud (Lee al. 2019). is known
P.I. function. It is also powerful operation that can project large input to smaller or arbitrary target shape. Previously, Perceiver (Jaegle et al. 202Tb) and Perceiver 10 (Jaegle al. (021a) have used this operation as replacement of the self-attention operation in Transformers reduce the complexity. However, cross-attention operation does not preserve data structure. In order to compensate for this, additional positional embedding is implemented to associate the position information with each input element.
Conversely, our work takes advantage of aotssnattnnnionss 's non-strrccurre-preeevving preserving nature to retrieve the P.I. style information. A key implementation detail is that position information should not be associated with the input tokens. Otherwise, there will be content leaks into the style path. To implement the style encoder, we follow previous work (Carion al. 2020
Lee et al. 2019) to learn
5

---

Published as a conference paper at ICLR 2022
dataset-shared prototypes Zo E Rmxd as seed vectors. Figure (a) shows the implementation details of the style encoder, which a stack of Lg cross-attention blocks, token mixing layers, and network (FFN) layers. The final output is the retrieved style S e Rmxd,
4.4 DECODER WITH LINK ATTENTION
In an autoencoder architecture, the decoder is responsible for reconstructing data from latent fea- tures. In learning content-style representation, the ability of the decoder limits the way content and style can be modeled. Therefore, decoder design becomes the key to the entire framework design.
In order support the bipartite graph representation of content and style, innovatively design link attention by unleashing the full potential multi-head attention. Unlike self-attention and cross-attention, link attention has distinct input for Q, K, and V. In the Retriever decoder, content tokens are queries and style tokens are values. The keys are set of learnable linking keys K I mdx which are paired with the style tokens represent the links in the content-style bipartite graph. Such design allows to retrieve content- sppeent-specifi style for data reconstruction, enabling fine-grained feature editing and interpretable style transfer.
Figure 4 (b) shows the implementation details of the Retrieve decoder. We first use convolu- tional layer directly after VQ to aggregate the contextual information as Y, = Conv(C). Then the style is injected into content for La times. In each round of style injection, we add self-attention layer before the link attention layer and append an FFN layer after it. The element order in K and S does not matter, long k; and s; are paired. The output of the decoder is then detokenized to reconstruct the raw data, whose difference with the input raw data is used as the reconstruction loss.
4.5 Loss FUNCTIONS
Our final loss function Laum consists of three components: the reconstruction loss Crees the VQ di- versity loss Q.Q. and the structural constraint loss Lsc (see implementation details in Appendix D
Ssum Areehree \vaLrq + AscLsc,
where Aree, \vQ, and Aso are hyper-paramette controlling the weights of the three losses.
5 EXPERIMENTS
We evaluate the Ret riever framework in both speech and image domains. Due the differences in tasks, datasets, and evaluation metrics, we organize the experiments in these two domains in two subsections. We use visualizations or quantitative results to: i) demonstrate the effect of content- style separation and the interpretability of content; ii) illustrate how the bipartite graph representation conveys the interpretability content to style and supports fine-grained style transfer.
5.1
SPEECH DOMAIN
5.1.1 TRAINING RETRTRVER FOR SPEECH SIGNALS
Retriever for speech signals is trained with the entire 44-hour CSTR VCTK Corpus (Veaux jt al. 2017) containing 109 speakers. We use the pre-trained CPC network (Rivière et al.) 2020) followed by depth-wise convolution perform tokenization. In Retriever, the content encoder is implemented by a VQ with two groups (G = 2) and V = 100. A truncated neighborhood cross- entropy loss is used as Lsc enforce the structural constraint, which is only applied to group #0. For the style encoder, we set the number of style tokens to 60. At the decoder, the content is first processed by ID depth-wise convolution with kernel size before being passed link attention. In the detokenization step, we resample the output log-Mel spectrum to 80Hz and feed it into Parallel WaveGAN vocoder (Yamamoto et al. 2020) to generate the waveform. We apply Ll reconstruction loss on log-Mel spectrum Lree. More details can be found in Appendix
5.1.2 VISUALIZATION OF CONTENT-STYLE REPRESENTATION
To understand how Retriever models the content, the style, and the bipartite graph between them, visualize the VQ codes and the decoder link attention map for 2.5s-long test utterance in

---

Published as a conference paper at ICLR 2022
Phonemes
Style token index
Mel- spectrum
VO Group
Phonemee
Group
Decoder
(a)
(b)
Figure 5: Visualization of (a) VQ codes, decoder link attention map, and the corresponding Mel
spectrum; (b) co-occurrence map between ground-truth phoneme and style tokens. Strong co-
occurrence is labeled with phoneme names, For higher resolution see Appendix B
Table I: Phoneme information probing.
Table 2: Comparison with other methods.
Settings
Frame
Frame w/ context
Methods
SV Accu
MOS
SMOS
CPC k-means
9.6%
9.5%
AutoVC Oian et al.
2019
17.9%
16510.11
1.9240.13
Group #0
48.1%
62.7%
AdaIN-VC
1.33
2019
46.5%
1.6+0.10
2.26+0011
Group
21.0%
45.9%
Tragment VC ( el
S2VC Lin
20216
89.5%
96.8%
34511012
3.18+0.12
3,36+0.15
3,54
Group #0, #1
52.3%
66.95
Recr 1ever
99.4%
0.44
3.8400.14
Figure (a). Each dot represents one token and the token hop is 10ms. Retriever encodes content with VQ. and we further impose structural constraint to VQ Group #0. We find that this group desired. Codes for adjacent tokens are highly con- sistent, and the changes of codes are well aligned with ground-truth phoneme changes. The VQ codes of group #1 exhibit a different pattern. They switch more frequently as they capture more detailed residual variations within each phoneme. We further study the interpretability of content by phoneme information probing (implementation described in Appendix F.3), which used show the information captured by the discovered content units (Chorowski et al. 2019). We measure the frame-level phoneme classification accuracy with or without context and compare with the baseline (CPC k-means). Results in Table [ show that the two groups of VQ codes discovered in our un- supervised framework align surprisingly well with the annotated phoneme labels. Notably, group #0 codes capture most of the phoneme information and group #1 codes are less interpretable. This demonstrates how the structural constraint helps to increase the interpretability of content.
of code demonstrates good interpretability
In decoder link attention, all the content tokens within the same phoneme tend to attend to the same style vector, while tokens from different phonemes always attend to different style vectors. We further visualize the co-oceurrence map between the ground-truth phonemes and the style vectors in Figure 5) (b) (calculation described in Appendix [B. We see that the phonetic content and style vectors have strong correlation. Interestingly, each style vector encodes the style of group of similarly pronounced phonemes, such as the group of 'M', 'N', and 'NG', This observation confirms that content specific style is actually realized in Retriever features.
5.1.3 ZERO-SHOT VOICE CONVERSION TASK
Zero-shot voice conversion task converts the source speaker's voice into any out-of-training-dan speaker's while preserving the linguistic content. We follow the setting of previous work to randomly select 1,000 test utterance pairs of 18 unseen speakers from the CMU Arctic databases (Kominek) Black 2004). Each conversion is given five target utterances to extract the style. Both objective and subjective metrics are used to evaluate conversion similarity (SV accuracy, SMOS) and speech quality (MOSNet score, MOS). More details of the four metrics are in Appendix F.22
Existing approaches fall into two categories: content-style disentanglement approaches QQian et al.) 2019 Yuan al.) 2021 Chou & Lee, 2019; Wu et al. 2020), and deep concatenative approaches (Lin et al. 2021b a). Methods in the first category use shared style embedding for all the content features while methods in the second category try to find the matching style for each content frame, resulting in high computational complexity. Our method provides a similar level of style granularity as the second class of approaches. But our bipartite graph representation is more compact and effec- tive it is capable of retrieving content- -npeett-ppecfiic style with a weight link attention operation. Table [ shows the comparison between our Retriever-basse method and four SOTA methods. The superior performance demonstrates the power of the content-style bipartite graph representation.
7

---

Published as a conference paper at ICLR 2022
5.1.4 ABLATION STUDY ON RETRIEVER
Based on the zero-shot voice conversion task, we carry out ablation studies on the VCTK + CMU Arctic dataset to better understand Retriever
Style Encoder. Unlike most previous work, which are limited to learning a single speaker style vector, our method can learn an arbitrary number of style tokens. When the number of style tokens increases from 10 and 60, the SV accuracy increases from 81 1.3% to 94.3% and 99.4%, showing the benefits of fine-grained style modeling.
Content Encoder. We study how the capacity of the information bottleneck affects the separation of content and style. Compared to the proper bottleneck setting (G' 2, V 100), speech quality significantly drops when the bottleneck "too narrow" (G = 1, V= = 10) and conversion similarity drops when the bottleneck is "too (G = 8, V 100), as shown in Table B This indicates content loss and style leakage when the bottleneck is too narrow and too wide, respectively.
Table 3:
Voice conversion quality at other content encoder and decoder settings.
Settings
SV Accu
MOSNet
AdalN-decoder
72.2%
2.89
Too narrow B.N.
99.8%
2.96
Too wide B.N.
90.5%
3.10
Proper B.N.
99.4%
3.12
Decoder. To illustrate the advantages of our link attention-based decoder, replace it with the traditional AdaIN module for experiment. As the AdaIN module only takes single vector as input, we flatten the 60 style tokens produced by Retriever into a very long vector. Results in Table show that, although the amount of information provided to the decoder is the same, AdaIN-based decoder has significant performance drop. This confirms the value of link attention.
5.2 IMAGE DOMAIN
The image domain one of the first domains to carry out the research on content-style disentangle- ment. Conventionally, shape is treated as content and appearance is treated as style. Interestingly, our unified definition of content and style aligns with the intuitive understanding of images.
5.2.1 TRAINING RETR TRVER FOR IMAGES
To tokenize the image, we downsample it by 4 times using a stack of convolutions. For VQ module, we set G 1, and V is dependent on the dataset. The convolution after VQ operation is of kernel
size 3 x 3. We detokenize the output tokens to images by convolutions and PixelShuffles (Shi et al. 2016). For the reconstruction loss Lrees we apply a typical perceptual loss (Chen Koltun 2017). For the structural constraint Lsc, we adopt a geometric concentration loss (Hung et al. 2019). We choose two commonly used datasets: Seleba-Wild (Liu et al.) (015) and DeepFashion (Liu et al. 2016). Images in Celeba-Witg are not aligned and each face has five landmark coordinates. The
full-body images from DeepFashion are used. See Appendix B for more details and experiments.
5.2.2 UNSUPERVISED CO-PART SEGMENTATION
Unsupervised co- co-part segmentation aims to discover and segment semantic parts for an object. It indi- cates whether the content representations are seman- tically meaningful. Existing methods (Hung al. 2019 Liu et al., 2021) rely heavily on hand-crafted dataset-specific priors, such background assump- tion and transformations. We choose SCOPS (Hung jet al., 2019) and Liu et al. (202T) as baselines and follow the same setting use landmark regression on Celeba- Wild dataset as a proxy task for evalua- tion. Please refer to the Appendix E.2 for details.
Table 4: Landmark regression results on CelebA-Wild K = V indicates the num- ber of foreground parts.
Method
K = 4
K = 8
SCOPS (w/o saliency) SCOPS (with saliency)
46.62
22.11
21.76
15.001
Liu et al. (2021)
15.39
12.26
Retriever
13.54
12.14
The content code of Retriever is visualized Figure 6 (a) for the Celeba- Wild dataset and in Figure X for the DeepFashion dataset. By setting small V in Retrievez, the encoded content only preserves the basic shape information as segmentation maps. Each content token is consistently aligned with meaningful part among different images. When compared the hate-of-the-art
8

---

Published as a conference paper at ICLR 2022
(a) Co-part
(b) Full
(c) Eye
(d) Jaw
Figure 6: Co-part segmentation and style transfer results Celeba-Wild Wild. Our method achieves
desired appearance transfer even on the unaligned dataset.
A
(a) Upper body
(b) Lower body
(c) Head
Figure 7: Part-level appearance manipulation DeepFashion. Our method can retrieve the correct appearance even with occlusion and large deformation.
methods, Thows, Retri everr is of the top performers this task, even without any hand-crafted transformations or additional saliency maps.
5.2.3 UNSUPERVISED PART-LEVEL STYLE TRANSFER
Part-level style transfer requires disentanglement of shape and appearance at part-level, which is particu- larly challenging in an unsupervised manner. Previous works (Lorenz
Style
Content
2019: Liu al., 2021) propose use one-one matching between shape and appearance, which is flexible and leads to unrealistic results. Our content-style bipartite graph separates content and style part-level and enables a more flexible content-specific style modeling. We can perform part-level style trans- fer by calculating the co-occurrence map between content and style tokens. See Appendix B for more details.
A
Both image-level and part-level transfer results are shown in Figure 6 7 and Retriever is capable of the explicit control of local appearance. Even without any supervision for the highly unaligned image pairs, method can transfer the appearance to a target shape with high visual quality. See Appendix E.4 for more results.
Figure 8: Shape and appearance trans- ferring on DeepFashion.
CONCLUSION
In this paper, have designed and implemented unsupervised framework for learning separable and interpretable content-style features. At the core of the proposed Retriever framework are the retrieval operations powered by the innovative use of multi-head attention. Cross-attention is used as the style encoder to retrieve style from the input structured data, and link attention is used the decoder retrieve content-specine style for data reconstruction. We have demonstrated that structural constraints can be integrated into our framework to improve the interpretability of the content. This interpretability is further propagated the style through the innovative bipartite graph representation. As result, the proposed Retri i lever enables couple of fine-grat fine-grained downstream tasks and achieves superior performance. As for the limitations, we have discovered experiments that different tasks different datasets need different model settings. We currently lack theoretical guidance for determining these settings, and we plan to work on in the future.

---

Published as a conference paper at ICLR 2022
REFERENCES
Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In CVPR, 2020.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vee: Self-supervised learning of discrete speech representations. In ICLR, 2020a.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In NewrIPS, 2020b.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. TPAMI, 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In AAAI, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and
Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017.
Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech repre- sentation learning using wavenet autoencoders. IEEE ACM Trans. Audio Speech Lang. Process., 2019.
Ju-Chieh Chou and Hung-yi Lee. One-shot voice conversion by separating speaker and content representations with instance normalization. In Interspeech, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Janek Ebbers, Michael Kuhlmann, Tobias Cord-Landwehr, and Reinhold Haeb-Umbach. Con- trastive predictive coding supported factorized variational autoencoder for unsupervised learning of disentangled speech representations. In ICASSP, 2021.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. ICLR, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Drew A. Hudson and Larry Zitnick. Generative adversarial transformers. In ICML, 2021.
Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming- King-Hsuan Yang, and Jan Kautz. SCOPS: self-supervised co-part segmentation. In CVPR, 2019.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver 10: A general architecture for structured inputs & outputs. CoRR, abs/2107.14795, 2021a.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira. Perceiver: General perception with iterative attention. In ICML, 2021b.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. In ICLR, 2017.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and S. R. Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In ECCV, 2018.
10

---

Published as a conference paper at ICLR 2022
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve: non- many-to-mane many voice conversion using star generative adversarial networks. In Spoken Language Technology Workshop, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
John Kominek and Alan W Black. The cmu arctic speech databases. Fifth ISCA workshop on speech synthesis, 2004.
Gihyun Kwon and Jong Chul Ye. Diagonal attention and style- GAN for content-style disen- tanglement in image generation and translation. In ICCV, 2021.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-inee invariint neural networks. In ICML, 2019.
Jheng-Hao Lin, Yist Y. Lin, Chung-Ming Chien, and Hung-yi Lee. S2VC: A framework for any-to- any voice conversion with self-supervised pretrained representations. In Interspeech, 2021a.
Yist Y. Lin, Chung-Ming Chien, Jheng-Hao Lin, Hung-yi Lee, and Lin-Shan Lee. Fragmentve: Any-to-any voice conversion by end-to-end extracting and fusing fine-grained voice fragments with attention. In ICASSP, 2021b.
Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Unsupervised part segmentation through disentangling appearance and shape. In CVPR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, 2016.
Chen-Chou Lo, Szu- Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin- Min Wang. Mosnet: Deep learning-baad based objective assessment for voice conversion. In Inter- speech, 2019.
Dominik Lorenz, Leonard Bereska, Timo Milbich, and Björn Ommer. Unsupervised part-based disentangling of object shape and appearance. In CVPR, 2019.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disen- tangling factors of variation in deep representation using adversarial training. In NeurIPS, 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In ICASSP, 2015.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In ICML, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David D. Cox. Unsupervised speech decomposition via triple information bottleneck. In ICML, 2020.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VAE-2.- NeurlPS, 2019.
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Rethinking content and style: Exploring bias for unsupervised disentanglement. In ICCVW, 2021.
Morgane Rivière, Armand Joulin, Pimrne-Emmanuel Mazaré, and Emmanuel Dupoux. Unsupervised pretraining transfers well across languages. In ICASSP, 2020.
Babak Salch and Ahmed M. Elgammal. Large-scale classification of fine-art paintings: Learning the right metric on the right feature. CoRR, abs/1 505.00855 2015.

---

Published as a conference paper at ICLR 2022
Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016.
Attila Szabó, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. disentangling independent factors of variation. In ICLRW, 2018.
Challenges in
Joshua B Tenenbaum and William T Freeman. Separating style and content with bilinear models.
Neural computation, 2000.
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In NeurIPS, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurPIS, 2017.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vetk corpus: English multi- speaker corpus for cstr voice cloning toolkit, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Dataset. Technical Report CNS-TR-201 1-001, California Institute of Technology, 201
Da-Yi Wu, Yen-Hao Chen, and Hung-yi Lee. VQVC+: one-shot voice conversion by vector quanti- zation and u-net architecture. In Interspeech, 2020.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Disentangling content and style via unsupervised geometry distillation. In ICLRW, 2019.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross B. Girshick. Early convolutions help transformers see better. CoRR, abs/2106.1489) 2021.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In (CASSP,2020
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Videogpt: Video generation using VQ-VAE and transformers. CoRR, abs/2104.10117. 2021. /t//ttts://xxxii.rr//// 2104. 10157
Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin. Improving zero-shot voice style transfer via disentangled representation learning. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2022
A BACKGROUD
Attention. Given n query vectors, each with dimension dg: Q E RRuxd an attention function Attn(Q, K, V) maps queries Q to outputs using m key-value pairs K Rmxd, V Rmxd,
QK
AQt K, V) = -sotmmax(
V
Vdg
where dot product QK R measures the similarity between queries and keys. softmax(-) normalizes this similarity matrix to serve as the weight of combining the m elements in V.
Multi-Head Attention, Vaswani al. (0117) propose multi-head attention, denoted as MHA(- It projects Q, K, V into h different groups of query, key, and value vectors, and applies attention function on each group. The output a linear transformation of the concatenation of all attention outputs:
O, Attn (Q0, KW. VW.), MHA(Q, K, V) concat O..... Wn
where W W.K E Rd, and W.V. x are the projection matrix for query, key, and value in the i-th head, wo Rddxdd is the linear transformation that fuses all the heads' outputs.
Self-Attention. Given an input array X E mxdd, a self-attention SA(-) or multi-head self-attention MHSA(-) operation takes X as query, key, and value:
SA(X) Attn(X X, xX
MHSA(X) MHA(X, X.
Cross-Attention Given input arrays X E nx Y E mmd,, cross-attention CA(:)) :) multi- head MHCA(...) operation takes X as query, and takes as key and value:
CA(X, Y) Attn(X, Y, Y)
MHCA(X, Y) MMH(X,,Y Y.Y)
Property: (multi-head) cross-altention operation is P.I. to Y, that is,
Vπ E PMM,MHCA(X -T (Y) = MHCA(X, Y).
where pm is the of all permutations of indices {1....,m) m]
Permutation Invariant Property of Style Encoder For the i-th style encoder block, we as- sume its input Zi-1 is P.I. to F. The output of the multi-head cross-attention layer is 2, MHCA(Z,-1. F). According to the property of multi-head cross-attention, mutation invariant to F. The following residue connection combines Z, with Z,-1, both of which are P.I. to F. Thus the combination is P.I. to F. The following token mixing layer and feed-forward layer don't 't take F as input. Because their inputs are P.I. F, their outputs are also P.I. to F. To summarize, if Z is P.I. to F, then Z, is also P.I. to F. It is obvious that Zu is P.I. to F since Zo fixed during the forward path. Therefore, the output of the whole style encoder is P.I. to F.
have 2, is per-
Vector Quantization. For efficiency, we use product quantization (Baevski et al., 2020a b). The quantized feature is the concatenation of G embeddings E1, e2, ..., EG E R247G which are looked up from the corresponding G codebooks E,, E2,... EG E Vxde/G where V is the number of entries in each codebook. To make VQ operation differentiable, Gumbel-softmax (Jang al.) 2017) is adopted: Firstly, each input token is mapped to logits i E jDCVV, Then, e, is calculated by weighted sum of codewords in codebook Eg. and the weight for adding the w-th code in group is given by:
exp((1g.v ngg.)/TT)
Wg.v
exp((),b ex_pex((1 ngg
where is temperature, Ng.e log( log(ug.))), and Ug. 4((0, 1) is independently sampled for all subscripts. We use the straight-througe estimator (Jang et al.) 2017). treating the forward and backward path differently. During the forward path, the one-hot version of the above weight
13

---

Published as a conference paper at ICLR 2022
one_hot (arg max; wg.k) is used, and in the backward path, the gradient of the original yg is used. To encourage all the VQ codes to be equally used, we use a batch-level VQ perplexity loss:
V
1 Σ exp(- Lve = GV
exp
Pg. g.)g)
(3)
GV
9=1
g=1
t=1
where P, indicates the probability calculated by softmax((l) averaged across all the tokens in batch, (-) calculates the entropy of discrete distribution.
B VISUALIZATION OF LEARNED CONTENT-STYLE BIPARTITE GRAPH
Retriever models the content-style bipartite graph by introducing the novel link attention mod- ule. The graph's edges are represented by the link attention map. To understand what content linked to which style, we gather the statistics of the link attention map as the content-style co- occurrence map. To show content information in 2-d map. the content is categorized by either the ground-tren -ruth labels, or the discovered parts or units. Specifically, the co-occurrence statistics of content category and style token 8, denoted Me,s. is calculated as follows:
Σ RED
A(2,8)((1(2) =c),
Mc,s
D
where D indicates the set composed of all the content tokens in the target dataset. A(x, s) is the link attention amplitude between content token r and style token s. I(-) is the characteristic function. It outputs if its input is true and outputs 0 if its input is false. ((x) gives the content category of token Jr.
For each style token, we consider the content category that shows the strongest co-occurrence with it to be the major content category that it serves. For the convenience of finding the pattern inside the co-occurrence map, we normalize on both content category axis and style token axis, then sort the style tokens so that their major content category index is in the ascending order.
In audio domain, we categorize content tokens using ground-trus truth phoneme labels and calculate the co-occurrence map using the first link attention. The content-style co-occurrence map is shown in Figure
Phonemes
15
20
30
40
35
0
10
30
40
Style taken index
Figure 9: Phoneme-style co-occurrence map of higher resolution.
In image domain, categorize content tokens using the discovered parts and calculate the occurrence map using the third link attention. The part indices together with content-style co- occurrence map shown in Figure I0
14

---

Published as a conference paper at ICLR 2022
Part index
2
1
Style token index
2
Part index
6
100
200
300
400
500
Style token index
Figure 10: Visualization of part-style co-occurrence map on Celeba- Wild and DeepFashion dataset.
15

---

Published as a conference paper at ICLR 2022
C STYLE SPACE VISUALIZATION
In order provide intuitive impression about what the styles decomposed by our framework are, we present visualizations of some style spaces produced by t-SNE (Van der Maaten Hinton 2008). We the clustered patterns in many style spaces which correspond to meaningful attributes like person gender and clothes color.
C.1 SPEECH DOMAIN
We visualize the style vectors extracted from 500 randomly-selected utterances of ten unseen speak- ers from Librispeech test-clean split. Figure I1 we show the latent space of four typical style vectors, which correspond to the style of nasal consonant (NG), voiceless consonant (T), voiced con- sonant (Z), and vowel (IH). Different speakers are labeled different colors. We see that different speakers are clustered into different clusters.
style NG
stle id phn_T
20
0-
-00
-20
-10-
-40
-00
-20
style id phn Z
style 59 phn
20-
0-
-40
-40
Figure 11: Style space t-SNE visualization in speech domain: utterances of the same speaker cluster together.
C.2 IMAGE DOMAIN
We examine the style vector spaces on Deepfashion dataset, and find that some style vector spaces are separately or collectively correlated with the human-defined attributes. For example, we find that the style vector #43 is highly correlated with the color of the clothes. Figure 12 shows the t-SNE visualization of this vector space, as well as some images each cluster. It is clear that images are clustered solely based the color of the clothes despite of various model identifications, standing poses, and styles of the dress.
As this dataset also has the gender attribute labelled, further look for the 'gender vector' in the decomposed style vector spaces. Interestingly, we find 14 vector spaces that are highly correlated to the gender attribute. We concatenate these 14 vectors and show the t-SNE visualization in Figure
16

---

Published as a conference paper at ICLR 2022
We see clear boundary between male and female models despite of the their color of clothes or standing poses.
styleid 43
10
-10
-10
-5
Figure 12: Style space t-SNE visualization in image domain: style vector #43 is 'clothes color vector".
30
10
0-
-20
-15
-10
5
10
15
Figure 13: Style space t-SNE visualization in image domain: ensemble of 14 style vectors iden- tifies gender.
17

---

Published as a conference paper at ICLR 2022
D STRUCTURAL CONSTRAINT
D.1 IMAGE DOMAIN
To make the content representation interpretable, spired by the observation that pixels belonging to the same object part are usually spatially concen- trated and form connected region, follow Hung al. (2019) use geometrical concentration loss as the structural constraint to shape the content representation to part segments. In practice, we denote the pixels assigned the first entry in the VQ codebook as background pixels and the other pixels as the foreground pixels.
apply additional structural constraint. In-
Given the logits E VV that assign each pixel to the parts (entries in VQ codebook), for foreground pixels:
h
Li -1 2= h
(4)
where cf and e are the coordinates of the u-th part center along axis h, and 24 a normalization term. o% e and 2 can be calculated as follows:
cx = Σ h Lo / c
Σ
w
Lo 2v:
(5)
h,w
Len
(6)
h,w
For the background pixels, different from previous methods using background loss (Liu et al. 2021 saliency map bound the region of interested object, do not add any constraint. Thus Lsc
Cig
D.2 AUDIO DOMAIN
To discover phoneme-like units speech, the discrete representation should switch fre- quently along the time axis. Therefore, we penalize the code switch based on the calculation of neighborhood cross-entropy.
LCE N-1 Σ CF #041) + CF(p++1
where N the length of the input sequence, CF(() calculates the cross-entropy between discrete probability and given label, p; denotes the predicted discrete probability before Gumbel-softmax, and 2, is the probast-rrobabiltyy code on position i.
However, Lox also penalizes the true phoneme switch, where the input acoustic feature really changes dramatically. To solve this problem, we use a truncated version of Log as our structural constraint:
Lsc = ytanh( LCE 7
(8)
where y is hyper-parameter indicating the truncation threshold. For the true phoneme switch, high LeE is expected, thus the loss will be truncated, while within the same phoneme, low Le is expected, and Lsc behaves like the original CE loss. As such, the truncated loss is more suitable for discovering phoneme-like units. In our experiment, we set = In(V) = In(100).
E IMAGE: ADDITIONAL DETAILS
IMPLEMENTATION DETAILS
We first show the tokenization module in Table the detokenization module in Table For the layer numbers, we set L., L,, La to 6, 3, 6, respectively. The hyper-parameter settings each style- encoder layer and decoder layer are shown in 7
and Table where d indicates the hidden
18

---

Published as a conference paper at ICLR 2022
dimension of the feed-forward network, and Mhad is the number of heads used in the multi-head attention modules. Specifically, for the decoder, we replace the FFN with Mix-FFN to improve the image quality. The Mix-FFN can be formulated
Fout MLP((GLUUCoosssse (ML((())))
where Tin is the input feature. The Gumbel-Softmax temperature r in VQ is annealed from 2 to minimum of 0.01 by factor of 0.9996 every update.
For the Celeba- Gale dataset, we resize the input to 128 x 128. For DeepFashion dataset, we resize the input 192 96, For the number of style tokens, in Table 9 Our model is implemented with Pytorch and trained on Nvidia V100 GPUs.
512. We summarize the training setting
Table 5: Tokenization module for images.
Conv x 24, stride
BatchNorm
ReLu
Table 6: Detokenization module for images.
Conv 3 3 x 24 48, stride
PixelShuffle(2)
BatchNorm
Conv x 3 x 48 x 48, stride
ReLu
ReLu PixelShuffle(2)
Conv 3 x 3 x 48 x 96, stride=
BatchNorm
Conv 3 3 12 x 12, stride
ReLu
ReLu
Conv 3 3 x 96 192, stride
Conv 1 1x 12x 3, stride
BatchNorm
ReLu
Conv 3 3 x 192 x 192, stride =
Table 7: Style encoder hyper-parameters.
Table 8: Decoder hyper-parameters.
Hyper-paramen
Value
Hyper-parameter
Value
dg 192
d & de
192
dFFN
768
dFFN
768
Thead
4
Thead
4
Table 9: Training setting.
Hyper-parameter
Value
Avea
λve
0.3
Xsc
or 7
optimizer
Adam (31 0.9, 82 0.999)
Learning rate
0.001
Batchsize
16
Iteration
150,000
E.2 DETAILS ABOUT EVALUATION ON CO-PART SEGMENTATION
For the evaluation on co-part segmentation, we follow the setting Liu et al. (00T)) and Hung et al. (2019). The proxy task for Celeba-Wild is landmark regression. We first convert part segmentations into landmarks by taking part centers as in Eq. S Then we learn linear regressor to map the converted landmarks to ground-truth landmarks and evaluate the regression error test data.
19

---

Published as a conference paper at ICLR 2022
E.3 ABLATION STUDY
Table 10: Ablation study on CelebA-Wild.
Method
Landmark regression error
Ours w/ bipartite graph
12.14
Ours w/ AdaIN
13.07
Ours w/o style branch
15.62
Ours w/ 16x downsampling rate
44.89
Ours w/ 8x downsampling rate
30.62
Ours w/ 4x downsampling rate
12.14
Ours w/ Aae
19.32
Ours w/ Ase 7
12.14
Ours w/ ABC 10
16.75
Ours w/ Aug =
15.98
Ours w/ Auy 0.01
13.58
Ours w/ Ang = 0.1
12.84
Ours w/ Ang 0.3
12.14
Ours w/ Apg =
21.60
Ours w/ Conv
12.14
Ours Petch
17.16
Do the style encoder and decoder impact the co-part segmentation? In this work, we find VQ module is suitable for unsupervised co-part segmentation, dataset. Here evaluate that if the style branch further helps the co-part segmentation. As shown Table 10 with a style branch, the discovered part segmentation is more consistent. Morcover, compare link attention decoder with AdaIN operation and find that our design is better for co-part segmentation task. This further verified that our content-style bipartite graph a powerful representation with high interpretability.
cluster center among the whole
On using different structural constraint weight. We further carry out experiments with different loss weight for structural constraint. Without the structural constraint, the parts learned seat- tered all over an image. As the weight for structural constraint getting larger, the part segmentation becomes tighter. With too large weight, the regions lose flexibilityy and fail to cover the parts.
On using different VQ diversity loss weight. We also test our model's sensitivity to Aug- As long Avg is set zero or too large (e.g., Aug 1), the model has fairly stable landmark regression eтror even when Aug is adjusted from 0.01 to 0.3.
(a) ABE
(b) ABE 4
(c) Asc =7
Figure 14: Visual ablation study on weight of structural constraint Asc-
On using different tokenization module. Different tokenization modules bring additional bias. Besides stack of convolution layers to tokenize the input image, we also experiment with the PatchMerge operation ViT (Dosovitskiy et al. 2021). which 4 x 4 convolution layer with stride 4 in help the preprocessing transformer blocks inside Retriever work better, which is also observed by concurrent work (Xiao et al. 2021)
As shown Table 10 using convolution layers the tokenization module can
20

---

Published as a conference paper at ICLR 2022
Ours w/ PatchMerge
Ours w/ Conv
Visual ablation study on tokenization module. Tokenizing the images using convolutions can help our model extract the content and style better, which leads to better transfer results.
Figure 15:
On using different downsampling rate. In this work, mainly 4x downsampling rate. For the input 128 x 128 image, we first tokenize it to 32 x 32. Here we experiment with additional different downsampling rates: 8x and 16x. addition the convolution layer for 4x setting, add additional convolution layers for further downsampling. We do not try 2x due to the limited computational resource. As shown Table for the selected downsampling rate, larger one results in worse performance. We also provide visual comparison in Figure 16
(a) 4x
(b) 8x
(c) 16x $
Visual ablation study downsampling rate. Larger downsampling rate leads to coarser co-part segmentation results.
Figure 16:
21

---

Published as a conference paper at ICLR 2022
Figure 17: More co-part segmentation results on Celeba- Willd dataset.
Figure 18: Co-part segmentation results CUB dataset. We set K = 4. Similar to previous work, due to the geometric concentration loss, we find struggles with distinguishing between orientations of near-symmetric objects, which is a common limitation.
E.4 MORE QUALITATIVE RESULTS
Co-part segmentation. We first provide more visualization on Celeba Wileba-Wildiii in Figure 17 Besides, we also apply our method on CUB dataset (Wah et al. 2011). CUB is dataset with 11, 788 photos of 200 bird species in total. This dataset is challenging due to the pose of the birds is of large diversity. We follow (Liu et al. 2021) select the photos from the first categories, and get about 200) photos. The results are shown in Figure
Part-level style transfer. Besides the results in the main paper, we also apply our method on a high-resolution dataset: CelebA-HQ. We resize the images 256 x 256. We show the part-level style transfer results in Figure [9 for mouth, nose, and eye.
Zero-shot image style transfer. We find model capable of generalizing into unseen image datasets. In Figure [0] we use our model trained on CelebA- to transfer artistic style from an artistic image. The transfer result keeps the pose and shape the content images (the first row), while adopting the tone and appearance of the style image (the first column on the left).
Visual comparison. We find our method enables more natural style transfer compared with the previous work. Here we show side-by-side visual comparison on Deepfashion dataset in Figure and 22 In image-level style transfer, the results of Lorenz al. (2019) suffer from significant deformation artifact. In contrast, such artifact is not observed in our results. In part-level style transfer, we show the results of head transfer. Again, our results are more natural and contain more image details.
22

---

Published as a conference paper at ICLR 2022
Reference
Mouth
Reference
Nose
Reference
Eye
Figure 19; Part-level style transfer results CelebA-HQ dataset for mouth, nose and eye. The resolution is 256 x 256.
Figure 20: Zero-shot image style transfer. Our model trained on CelebA- Wild but can general- ized to artistic images. The artist image (left) is obtained by AdaIN (Huang & Belongie 2017 from CelebA-Will (Liu et al. 2015) and WikiArt (Saleh & Elgammal, 2015) dataset.
23

---

Published as a conference paper at ICLR 2022
(a) Ours
(b) Lorenz et al. (2019)
Figure 21: Visual comparison on Deepfashion dataset: image-level style transfer. Our results are more natural than the baseline. Results of Lorenz al. (2019) are cropped from their official web- site: https: btttss:/ccopviii github. www.usuuuuvvviiiccocoo www.-iseedddddddddddnnnnnni disentangling
(a) Ours
(b) Lorenz et al. (2019)
Figure 22: Visual comparison on Deepfashion dataset: head style transfer. Our results are more natural than the baseline and contain more image details.
24

---

Published as a conference paper at ICLR 2022
F AUDIO: ADDITIONAL DETAILS
F. IMPLEMENTATION DETAILS
For the tokenization module, we follow Lin et al. (2021a), using s3prl toolki to extract the CPC feature, and use a depth-wise convolutional layer with kernel size 15 after the CPC feature. The depth-wise convolutional layer is trainable, while the CPC model is fixed during training. For the detokenization module, we use publicly available implementation of Parallel WaveGAN pretraineed on LibriTTS, Tee Tæg, Ltg are d. = 512. Following Qian et al. (2020), we add auxiliary quantized normalized fo feature into con- tent for better voice quality. The Gumbel-Softmax temperature is annealed from of 0.01 by factor of 0. 0.9996 at every update. The hyper-parameter settings of each style-encoder layer and decoder layer are shown in Table [ and Table [2 where dFFN indicates the hidden dimension of the feed-forward network, and Rhead is the number of heads used in the multi-head attention modules. The decoder output converted 80-d log-Mel spectrum with an MLP whose hidden dimension is 4096. The training hyper-paramete setting is listed in Table 13 Each training sample random 4-second audio clip sampled from the training dataset. For utterances that are shorter than 4s, we pad zero at the end of the utterance. The code is implemented with Pytorch, and training takes hours on 4 Nvidia V100 GPUs.
0, 3, respectively. The dimension of content representation is
minimum
Table 11: Style encoder hyper-parametern. Hyper-parann Sarere-prrmettr
Table 12: Decoder paper-parametess. Hyper-parameter
Value
Value
d,
192
d
512
dPFN
512
dFFN
2048
Whead
4
Thead
8
dropout
0.1
dropout
0.1
Table 13: Training setting.
paperr-arraeeter
Value
Arec
\vQ
0.3
Asc
0.1
Optimizer
Adam (31 0.9, 32 = 0.999)
Learning rate
0.004
Learning rate schedule
power (p 0.3, warmup-steps 625
batch-size
120
epoch
50
F.2 EVALUATION METRICS OF ZERO-SHOT VC TASK
For objective metrics, conversion similarity measured by Resemblyzer speaker verification system by calculating speaker verification accuracy (SV Accu) between converted speech and correspond- ing target utterance, as done in the previous work (Lin et al.] (2021b). The conversion considered successful if the cosine similarity between Resemblyzer speaker embedding of converted speech and target utterance exceeds a pre-defined threshold. The threshold is decided based on equal error rate (EER) of the SV system over the whole testing dataset. The SV accuracy is the percentage of suc- cessful conversions. The objective speech quality metric is estimated by MOSNet (Lo et al. (019)). takes converted speech as input and outputs a number ranging from 1 to as the measurement of speech naturalness. Both metrics are higher the better.
hutees///gttubb.cm////////////
htrass//iiabbb..coo/knaaaaaaaaa laaaa e WaveGAN
https://gittubb.com/reeeett/
yzer
25

---

Published as a conference paper at ICLR 2022
For subjective metrics, conduct two tests to calculate the opinion score of conversion similarity (denoted as SMOS), and speech quality (denoted as MOS). For conversion similarity, each subject is asked to listen to the target utterance and converted utterance, and judge how confident they are spoken by the same speaker. For speech quality, each subject is asked to listen to utterances that are randomly chosen from converted speech and real speech, and determine how natural they sound. Both test results are scored from to 5, the higher, the better. We randomly sample 40 utterances from the test set for both tests. Each utterance is evaluated by at least 5 subjects. The score is then averaged and reported with 95% confidence intervals.
F.3 PHONEME INFORMATION PROBING: TRAINING AND TESTING
We conduct phoneme information probing to test how much phoneme information is encoded in the learned discrete content tokens, as to demonstrate their interpretability. We experiment with two settings, one considers the contextual information (denoted as "Frame w/ context"), ) and the other considers only single-frame information (denoted as "Frame"). For these two settings, Id convolutional layer with kernel size 17 or a linear layer is used as the probing network respectively, to predict ground-truth phoneme label. For the input of the probing network, each frame is represented by the one-hot vector of the corresponding code. For the experiment involving two groups of code, the one-hot vector of both groups are concatenated on the channel axis. For the k-means clustering of CPC feature, the cluster number is set to 100, which is the same as the number of codebook entries per VQ group. We train the probing network on LibriSpeech train-clean- 100 split and test on LibriSpeech test-clean split. Adam optimizer is used with 81 0.9, 32 0.999. Learning rate and batch size are set to 0.00005 and 30, respectively. Each training and testing sample a random 2-second segment from the dataset. We drop the utterance that is shorter than 2 seconds. In Table we report the test accuracy after the convergence of the training.
F.4 DETAILS IN SYSTEM COMPARISON
We choose four SOTA systems for comparison. Two content-style disentanglement approaches
are AutoVC QQian et al. 2019 and Ad (Chou & Lee 2019 Two deep concatenative
approaches are Fragment VC (Lin al. 2021b 6 and S2VC (Lin et al. 2021a All these methods
have officially released their code, together with the pretrained models also on VCTK dataset. We adopt these models and test them on our randomly sampled 1000 conversion pairs in CMU Arctic dataset.
F.5 INFERENCE SCALABILITY ON ZERO-SHOT VC TASK
Table [4 shows the inference scalability of method on VCTK + CMU Arctic dataset. "# target" indicates the number of available target utterances for inference. We see that Retrieve performs quite well in the case that only one target utterance is given. As the available target utterance increases, the conver- sion similarity keeps increasing up to 99.6%, and it has no negative effect on speech quality, indi- cating the extracted style becomes more accurate when seeing more target samples.
Table 14: Inference scalability
target
2
10
SV accu (%)
95.1
98.7
99.4
99.4
99.6
MOSNet
3.12
3.13
3.12
3.12
3.13
F.6 PERFORMANCE ON LIBRISPEECH DATASET
LibriSpeech dataset (Panayotov et al. 2015) is more diverse than VCTK CMU Arctic in the aspect of emotion, vocabulary, and speaker number, thus closer to real-world application. In this experiment, the whole train-clean- 100 split containing 251 speakers used for training, and test- clean split containing 40 speakers is used for testing. When testing, the conversion source-target pairs are built as follows: for each utterance in the test set, we treat as the source utterance, and
26

---

Published as a conference paper at ICLR 2022
assign it one target utterance that is randomly sampled from the test set. In this way, each utterance in the dataset serves as the source for one time.
Objective similarity metric is measured for using different style token numbers, as shown in Figure 23 The conver- sion similarity increases when using more style tokens, and the performance saturates at nearly 100%. To be spe- cific, achieve 98.4% SV accuracy and 3.13 MOSNet score when using 60 style tokens, demonstrating that our method can generalized into more complicated scenar- ios and potentially can be used in the real-world applica- tion.
0.95
Accu
0.90
20
40
60
80
100
#style token
Figure 23: Performance on LibriSpeech dataset.
F.7 LOss WEIGHT ABLATION STUDY
Table 15: Loss weight ablation study on VCTK + CMU Aretic dataset.
Method
SV Accu (%)
MOSNet
Ours w/ Àsc
98.9
3,01
Ours w/ Asc 0.002
98.5
3.10
Ours w/ Asc 0.01
99.0
3.11
Ours w/ Asc 0.02
98.5
3.12
Ours w/ Asc 0.05
99.0
3.10
Ours w/ Asc 0.1
99.4
3.12
Ours w/ Asc 0.2
99.7
3,07
Ours w/ Asc 0.5
99.3
3.12
Ours w/ λvQ = 0
99.4
2.94
Ours w/ λvQ 0.01
99.5
3.08
Ours w/ AvQ = -0.3
99.4
3.12
Ours w/ λvQ 2.0
99.5
3.08
To test the effectiveness of structural constraint and VQ diversity loss, we do ablation study on the corresponding loss weights, λvQ and Àsc The SV accuracy and MOSNet scores achieved at different parameter settings shown in Table 15. We observe that the absence of either loss term leads to noticeable speech quality (MOSNet score) drop. We further test the model's sensitivity to these two loss weights and we able empirically conclude that our framework is not sensitive to the selection of these two syper-parammterr When Àsc increases from 0.002 to 0. (by 50 times), SV accuracy only fluctuates by +0.45% and MOSNet score only fluctuates by +0.01. When λvy increases from 0.01 to 2.0 (by 200 times), SV accuracy only fluctuates by +0.05% and MOSNet score only fluctuates by =0.02.
In a nutshell, both structural constraint and VQ diversity loss are necessary, and our model is not sensitive the selection of the corresponding loss weights in reasonably large range.
27


Reviews:
[{"review_id": "saC3Xc5edFX", "reviewer": "Reviewer_WCnU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Interesting idea of content and style disentanglement; validated on both image and audio domain.\n# Strength:\n1.\tThe paper proposes a new way to define style (for a given dataset) using permutation invariant (P.I) network. This is a good attempt to have a common definition of style.\n2.\tThe network architecture is novel although using existing components such as VQ network and attention network. The idea of disentangling and link attention are what make the paper interesting.\n3.\tThe model is domain-agonistic – demonstrated to work well in audio and image domain\n4.\tThe paper demonstrates various downstream task in each domain. The part style transfer is impressive although there are some artifacts. Nice that the authors provided audio sample to play in supplementary file.\n\n# Weakness/Suggestion/Queries \n\nThe paper proposes to separate out style representations. But it lacks visualization of how actually each individual style visually look like. It may be that these styles are actually relates to attribute of the dataset which for some datasets (e.g. DeepFashion, CUB birds) already defined. It would be interesting to cluster these style features and see the discovered styles from a given dataset.\n\nOverall, there seems to a lot of hyperparameters to tune and may be that one must have good understanding of dataset for proper hyperparameter settings. Details as follows:\n\n1.\tHow is $r_{\\theta}$ implemented? It seems this is the linking attention network. Could you please include some discussion with guiding equations at least in the supmat.\n2.\tContent Encoder: It is stated  the spatially adjacent tokens are forced to share the same VQ code. If so, how does this work when there are two different objects (parts) in the adjacent tokens (adjacent patches in images)?\n3.\t$\\mathcal{Z}_o \\in \\mathbb{R}^{m \\times d_s} $: What is a suitable value for m? Does the value of m relates to the number of styles the network can represent?\n4. The total loss is sum of three loss terms with $\\lambda$ weighting parameters. How sensitive is the model (training/convergence) to the hyper-parameters $\\lambda$ s? If the choice of these parameters slightly off, do the network perform still reasonably?\nAny training strategy used to deal with discrepancies brought by these parameters? How do you train the network in different stages e.g., adding more loss terms in subsequent stage?\n\n5.\tHow do you come up with G=2 and V=100 VQ is sufficient to capture all content information for audio dataset? Any heuristics for this, or perform rigorous parameter search?\n6.\tDo you do manual observation to associate content codes with parts in image domain? It is stated that V is dependent on image datasets. Does this mean that you set V to the number of parts you want to segment out from images?\n7. It is not clear how the authors generate: (a) Codebooks for content (b) Prototype vectors for style?\n\n", "summary_of_the_paper": "The paper proposes a novel idea of disentangling the content and style of structured data by treating style as permutation invariant information. It adopts VQ network for content encoding, and Cross-Attention for Style and Linking Attention at decoder. It is shown to be domain agonistic - worked well in image and audio domain.", "summary_of_the_review": "The idea is interesting and seems novel, and the method is domain agonistic that may be applied to other domains. Experiement conducted are convincing, I am leaning to accept the paper, provide the minor concerns raised are resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Q5tdroJ_4Ym", "reviewer": "Reviewer_c2Ut", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nS1: This paper addresses an important problem -- content and style decomposition representation learning and proposes an unsupervised framework to learn content-style representations.\n\nS2: The proposed model performs relatively well on both speech and image domains.\n\nWeaknesses:\n\nW1: It is difficult to follow the paper some notations lack details. For example, what are the prototypes and link keys in Figure 3 and how to obtain them? For speech and image domains, how to obtain the input of the model?\n\nW2: Lack of ablation study on hyper-parameters in Eq. (1). What are the roles of the three items in Eq(1)?\n\nW3: It is unclear how the model avoids or mitigates information leaking. It seems that a perfect model should avoid the style leak while achieving perfect reconstruction (shown in Figure 2 (c)).\n", "summary_of_the_paper": "This paper proposes an unsupervised framework to learn content and style representations of structured data, such as texts, images and speeches. The task is interesting to me and decomposition representation learning is important. In addition, a bipartite graph is proposed to model the link between content tokens and style tokens, making the model more interpretable. The proposed model performs relatively well on both speech and image domains.", "summary_of_the_review": "This paper addresses an important problem and proposed a framework to learn content and style representations. The experiments show that the performance of the proposed model is relatively well. But some details are missing, so it is difficult to follow the paper and it is unclear the roles of the loss functions and how the model avoids information leaking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "GfQBbMJMw6r", "reviewer": "Reviewer_1d43", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n- S1. The proposed method is multimodal and can serve multiple modalities with a broad definition of style and content. The paper provides results on speech and vision data.\n- S2. The proposed method is unsupervised, so it does not need annotation on style/content for the training data. This is very useful, as the boundaries on style and content are not straightforward and challenging to annotate.\n- S3. The paper is clear and easy to read. \n- S4. Experiments are technically sounds.\n\n*Weaknesses*\n- W1. The definition of style as \"the information that is not affected by the permutation of tokens\" is a bit counterintuitive — no justification or examples of why the paper chooses this, and no other definitions are provided. \n- W2. The definition used in the paper does not correspond to the traditional definition of style and content in computer vision, which has been (and still is) a widely studied problem. This creates some confusion, especially in the image domain section.\n- W3. After multiple examples referring to text as a potential target modality for this framework, no experiments are conducted on NLP datasets.\n- W4. Additionally, for the image domain experiments, it would have been interesting to see results on art datasets, a field in which the problem of style and content disentanglement has been severally studied.\n- W5. In the image domain experiments, there is no comparison to other methods other than for the proxy task of landmark regression. As visual results on style transfer are provided, it is expected to compare against other methods to see the potential of the proposed model against current technology.", "summary_of_the_paper": "This paper proposes a framework for learning disentangled representations of content and style in an unsupervised way. The paper presents a slightly uncommon definition of content and style, which serves as a foundation for the proposed methodology and experiments. Experiments are conducted on speech and image datasets. ", "summary_of_the_review": "The idea of the paper is interesting and potentially useful for the community. However, the experimental results seem limited for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for learning disentangled representations of content and style in an unsupervised way, using a permutation invariant network. It adopts VQ network for content encoding, and Cross-Attention for Style and Linking Attention at decoder. It is shown to be domain agonistic, working well in image and audio domain. Experiments are conducted on speech and image datasets.

The paper is recommended as an accept (weak) to ICLR. The reviewers have given detailed feedback and suggestions -- please address them in the next revision of the paper.
Decision: Accept (Poster)