Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints
Authors: Yuxuan Wu
Ziyu Wang
Bhiksha Raj
Gus Xia
Venue: ICLR 2025 Poster (2025)
Keywords: Unsupervised learning
Interpretable representation learning
Emergent knowledge
Self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose to use a meta-level inductive bias about content and style to learn their disentanglement, showing good out-of-distribution performance and enabling the emergence of interpretable content symbols that align well with human knowledge.

Abstract: We contribute an unsupervised method that effectively learns disentangled content and style representations from sequences of observations. Unlike most disentanglement algorithms that rely on domain-specific labels or knowledge, our method is based on the insight of domain-general statistical differences between content and style --- content varies more among different fragments within a sample but maintains an invariant vocabulary across data samples, whereas style remains relatively invariant within a sample but exhibits more significant variation across different samples. We integrate such inductive bias into an encoder-decoder architecture and name our method after V3 (variance-versus-invariance). Experimental results show that V3 generalizes across multiple domains and modalities, successfully learning disentangled content and style representations, such as pitch and timbre from music audio, digit and color from images of hand-written digits, and action and character appearance from simple animations. V3 demonstrates strong disentanglement performance compared to existing unsupervised methods, along with superior out-of-distribution generalization and few-shot learning capabilities compared to supervised counterparts. Lastly, symbolic-level interpretability emerges in the learned content codebook, forging a near one-to-one alignment between machine representation and human knowledge.

Published as a conference paper at ICLR 2025
UNSUPERVISED DISENTANGLEMENT OF CONTENT AND STYLE VIA BARIANCE-INVARIANE CONSTRAINTS
Yuxuan Wu' Ziyu Wang Bhiksha Raj*
Gus Xia
b Mohamed bin Zayed University of Artificial Intelligence
# New York University Shanghai
t Carnegie Mellon University
ABSTRACT
We contribute an unsupervised method that effectively learns disentangled content and style representations from sequences of observations. Unlike most disentangle- ment algorithms that rely on domain-specific labels or knowledge, our method is based on the insight of domain- komain-general statistical differences between content and style - content varies more among different fragments within sample but main- tains an invariant vocabulary across data samples, whereas style remains relatively invariant within a sample but exhibits more significant variation across different samples. We integrate such inductive bias into an encoder-decoder architecture and name our method after V3 evariance-versury verssr-iivviuiiannneee invariance). Experimental results show that V3 generalizes across multiple domains and modalities, successfully learning disentangled content and style representations, such as pitch and timbre from music audio, digit and color from images of waritwritten digits, and action and character appearance from simple animations. V3 demonstrates strong disen- tanglement performance compared to existing unsupervised methods, along with superior nut-of-distribution generalization under few-shot adaptation compared to supervised counterparts. Lastly, symbolic-level interpretability emerges in the learned content codebook, forging a near one-to-one alignment between machine representation and human knowledge 1
INTRODUCTION
Learning abstract concepts is an essential part of human intelligence. Even without any label supervision, we humans can abstract rich observations with great variety into a category, and such capability generalizes across different domains and modalities. For example, we can effortlessly perceive picture of a "cat" captured at any angle or set against any background, we can perceive the symbolic number "8" from an image irrespective of its color or writing style variations, and we can perceive abstract pitch class "A" from an acoustic signal regardless of its timbre. These concepts form the fundamental vocabulary of our languages be they natural, mathematical, or musical and underpin effective and interpretable communication in everyday life.
Our goal is to emulate such abstraction capability using machine learning. We choose a content-spp representation disentanglement approach we believe that representation disentanglement offers a more complete picture of abstraction torrcppioncconceps that matter more in communication, such as an "8" in written phone number or a note pitch "A" in a folk song, are usually perceived as content while the associated variations that often matter less in context, such as the written style of a digit or the singing style of a song, are perceived as style. In addition, content is usually symbolized and associated with rigid labels, as we need precise control over it during communication. E.g., to write "8" as "9" in a phone number or to sing an "A" as "B" in a performance can be fatal error. In comparison, though style can also be described discretely, such as an "italic" writing or "tenor" voice, a variation over usually much more tolerable.
'Code is available at http::/s//gtttoo....../e/eeeeeee Demo can he found at httpps:///--oontettttttttee
www.hhuub.oo/v3deeeee demo/

---

Published as a conference paper at ICLR 2025
Variant within sumple
Content
98811247530
8214069537
Invoriant vocabulary
Sumples
5477138092
Samples
Samples
Fariont
across samples
6583720941
across samples
4158723690
3648025971
Fragments
Fragments
Fragments
Style
Invoriant umong frugments within sample
Figure An illustration of the variance-verssu-innan -nvarrianne constraints in content and style. Here, content refers to the symbols. Each row represents data sample, which is divided into multiple fragments along columns. Each fragment contains one content-style pair. For example, digits (c.g.. 8, 6) can represent content, while colors (e.g.. orange, brown, teal) represent style.
In the machine learning literature, significant progress has recently been made in content-style disentanglement for various tasks, including disentangling objects from backgrounds (Hong et al.
2023). characters from fonts (Liu et al. 2018: Xie al., 2021), pitch from timbre
(Lu al., [019
Bonnici et al. 2022). and phonemes from speaker identity QQian et al. [019 Li al.
(2021). However,
most existing models are either limited to specific domains (Yingzhen and Mandt 2018 Bai et al.
2021 Luo et al. 2022) rely heavily on camain-speccfic knowledge as implicit supervision. The
supervision forms can explicit content or style labels (Liu et al.,
2017 Zhu et al. [017]] Park
al. 2020 Karras et al. 2019 Choi et al., 2020 Bonnici et al. [2022
Patashnik et al. 202 Kwon
and Yel (022), pre-trained content or style representations (Qian et al.
2020b 2019), or paired data showcasing the same content rendered in different styles vice versa (Isola al. 2017 Sangkloy
ctt al.] (017). In addition, the disentangled representations often fell short in generalizing to new contents or styles, and they lack interpretability a symbolic level and do not align well with human
perceptions (Zhang et al. 2021 Nauta et al. 2023).
To address the aforementioned challenges, achieving more generalizable and interpretable disentan- glement in an unsupervised manner, we introduce V3 (variance- (ttrrsuus--variacee) V3 disentangles content and style by leveraging meta-level prior knowledge about their inherent statistical differences. As shown in Figure our design principle is based on the observation that content and style display distinct patterns of variation content undergoes frequent frequett changes within different fragments of sample yet maintains a consistent vocabulary across data samples, whereas style remains relatively stable within sample but exhibits more significant variation across different samples.
In this paper, we adopt the quctor-quattzzed autoencoder architecture and incorporate variance- versus-invann insaruiivvre constraints to guide the learning of latent representations that capture content-style distinctions. We demonstrate that V3 effectively generalizes across distinct areas: disentangling pitches and timbres from musical data, disentangling numbers and ink colors from images of digits, and disentangling character actions and appearances from game video clips. Experimental results show that our approach achieves more robust content-style disentanglement than unsupervised baselines, and outperforms even supervised methods in distiiiiiibttiionn (OOD) generalization and few-shot learning for discriminative tasks. Lastly, symbolic-level interpretability emerges with near one-to-one alignment between the vector-quantized codebook and human knowledge, outcome not yet seen in previous studies, In summary, our contributions are as follows:
Unsupervised content-style disentanglement: We introduce V3, an unsupervised method leveraging meta-level inductive bias to disentangle content and style representations, without requiring paired data, content or style labels, domain-speciin assumptions.
Out-offdistributioo generalization: As a result of successful content-style disentanglement, V3 shows better out-of-distre distrribiitt generalization capabilities compared to supervised methods in shw-shot settings, that is, recognizing content when presented with only a few examples of unseen styles.
2

---

Published as a conference paper at ICLR 2025
Emergence of interpretable symbols: Given the availability of semantic segmentations, V3 can foster the development of interpretable content symbols that closely align with human knowledge.
2 RELATED WORK
The content-style disentanglement as well as the related style transfer problem has been well explored in computer vision, especially in the context of image-to-image translation. Early works mostly
require paired data of the same content with different styles (Isola et al., 2017, [Sangkloy et al., (2017)
until the introduction of domain transfer networks that can learn style transfer functions without
paired data (Zhu et al., [017 [iu et al., [2117] Taigman et al.) 2016; Bousmalis et al. 2017 Park et al. 2020 Choi et al. 2018; 2020] Karras et al. [ et al.) 2022a bj. Although these methods are
unsupervised in the sense that they do not require paired data, they still require concrete labels of
styles to identify source and target domains, and there are no fully interpretable representations of either content style.
similar trajectory of research has also been followed in other domains including speech Qian et al. (2019); Kameoka al. (2018); Kaneko et al. (20T9); Wu al. (2023) and music Lu et al. (2019);
Bonnici et al. (2022); ; Luo et al. (2022): Lin et al. (2023); Zhang et al. (2024); Lin et al. (2021). To
mitigate the requirement for supervision, some methods utilize domain-
knowlede and have
achieved better disentanglement results, including X- X-vectors of speakers Qian et al. (2019 2020a).
the close relation between fundamental frequency and content audio Qian al. (2020a b). or
pre-defined style or content representations Yang al. (2019); Wang et al. (2020) 2022).
Pure unsupervised learning for content and style disentanglement has not been well explored. Notable attempts include adversarial training- based methods (Chen et al., 2016 Renn et al., (021), mutual
information neural estimation (MINE) (Belghazi et al. 2018 Poole et al.] 2019 Tjandra et al. 2020a Zhang and Dixon 2023), and low-dimensional representation learning with physical symmetry (Liu et al.; 2023). But these methods often suffer from the training stability issue or have not been proven to be domain domain-general. Other unsupervised methods sequential data, such Disentangled Sequential Autoencoder (DSAE) and its variants, leverage the nature of content and style to learn their representations at different scales, but their applications are limited purely sequential data with a static style (Hsu et al. 2017 Yingzhen and Mandt 2018 Bai et al. 2021
Luo et al. 2022
2024 Yin al. 2022).
A technique often associated with learned content is vector quantization (VQ) (Van Den Oord et al. 2017). Recent efforts have built language models on top of VQ codes for long-term generation, indicating the association between VQ codebook and the underlying information content Yan et al.
2021; Tan et al. 2021; Copet et al., 2024) Garcia et al., [023] Tjandra al., 2020a b Vali and Bäckström 2023). A noticeable characteristic of these studies is the use of large codebooks, which limits the interpretability of representations. We borrow the idea of small codebook size from categorical representations (Chen et al. 2016) Ji et al.) 2019), targeting more concise and unified content code across different styles, while keeping the high-dimensional nature of VQ representations.
METHODOLOGY
Considering dataset consisting of data samples, where each sample contains fragments, we aim to learn each fragment's content and style representation with the inductive bias illustrated in Figure I Intuitively, the fragments within each data sample have relatively eequeently-changing content and relatively stable style. For different data samples, their style exhibits significant variations and their content more or less keeps consistent vocabulary. In the following, we first introduce the autoencoder architecture V3 is built upon, then the variability statistics to quantify the changing patterns of content and style, and the proposed variance- versus- wwwerveeersrssiiiaieeee invariance constraints.
3.1 MODEL ARCHITECTURE
The model architecture of V3 is illustrated in Figure 2 Let X
(x.y)xx be the dataset, where Zij corresponds to the j-th fragment the i-th sample. We use an autoencoder architecture to learn the representations of m.y. The encoder encodes the input data mi to the latent space, which is split
3

---

Published as a conference paper at ICLR 2025
Codebook
zi2
VQ
AN1
Zij
Lonntent
Vi
<<
iij Encoder
Decoder
L Bij
Cuampee
Pragmeent
VA
Z
Catyle
#11
#12
Figure 2: The model architecture of V3. Left: The autoencoder has two branches for content and style respectively, where the content branch has VQ layer at the encoder output. Right: the V3 constraints, where Kouble-dashed arrows represent measuring the variability by vx(+). and solid arrows represent taking the average.
into to zij and z.y. We use vector quantization as the dictionary learning method for content. Every content representation zij is quantized to the nearest atom in codebook of size K zi.. The decoder integrates EG and zīy and reconstructs the fragment æij. The overall loss function is the weighted sum of three terms:
= + aLvq + 3Lv3-
Here, Lree is the reconstruction loss of X and Liq is the VQ commit loss (an Den Oord et al. 2017)
Lree x L
Diy
E.y 11
(2)
j=1
L
| sg(2i,)ll2.
N
X
where sg(.) is the stop gradient operation of the straight-through optimization. The final term Lv3 the proposed regularization method to ensure unsupervised content-style disentanglement, which introduce in the rest part of this section. (For more details of the model architecture and data representations, we refer the readers to Appendix B
3.2
VARIABILITY STATISTICS
We define four statistics to measure the degree of variability in accordance with the four edges of Figure These statistics are based on a backbone variability measurement 2 (.), where & represents the dimension along which variability computed. In this paper, define vi(()) (-) the mean pairwise distance (MPD). Formally, for a vector z of length D.
D Σ
( = MPD2
1
Σ
||
23112-
D(D - i=1j=l.jzi
The motivation for using MPD is that it is more sensitive to muuli-ppat distributions than standard deviation, which is preferred when learning diverse content symbols a sample. We compare different choices of es (.) in Appendix D
4

---

Published as a conference paper at ICLR 2025
Content variability within sample (VF). We first compute the variability of content along the fragment axis and take the average along the sample axis. The value is the average of content codes before and after vector quantization:
N 2N
V
(= +
(2
(5)
2N
Content variability across samples (V.)) Theoretically, we aim to measure the consistency of codebook usage distribution along the sample axis, which is not differentiable. In practice, we compute the center of the content code along the fragment axis and measure the variability of the centers along the sample axis. It serves proxy of codebook utilization. Also, we consider both content codes before and after vector quantization:
V = 2
Σ
+
Σ
L
Style variability within sample (vF). We compute the variability of style representations among fragments and take its mean across all samples:
vp
(z)))
Style variability across samples (V.). We compute the average style representation along the fragment axis and measure its variability along the sample axis:
=
(8)
3.3 VARIANCE- VERSUS-INVAIINNGG (V3) CONSTRAINTS
With the variability statistics, formalize the general relationship between content and style along the sample or fragment axis:
Content should be more variable within samples than the aggregated content across samples, i.e., V/ > V.
Style should be more variable across samples than within samples, i.e., V. >> V..
Within a sample, content should be more variable than style, i.e, 1/8 > 1/.
Across samples, style should more variable than content, i.c., 1 > V.
We quantify the above contrasts regularization terms, using the hinge function back-proe mack-propagation when the ratio between two variability statistics reaches a certain threshold r > 1,
off gradient
which stands for relativity (Bardes et al. 2022):
Ceontent = max(0,
)
(VF > V9)
(9)
r Ve
Latyle max(0, 1
(V. > Vp)
(10)
r Vp
V
Lirugment max(0,
(V/ > vp)
(11)
Vp
Lsample = max(0, 1
(V*
> V
(12)
Ve
We obtain the V3 regularization term (used in Equation 1 by summing up the four terms:
Lv3 Leonteut + Laayle + Liragment + Lsample-
(13)
5

---

Published as a conference paper at ICLR 2025
Table Evaluation of digit and color disentangiement on PhoneNums using latent retrieval. Values are reported in percentage.
Content
Style
PR-AUC
Best Fl
PR-AUC
Best Fl
Method
K
2"
2"
x"
10
83.2
12.8
84.1
18.5
14.9
95.4
22.6
91.0
93.0
11.6
92.9
18.2
11.9
93.9
22.7
40
86.3
10.9
83.4
18.0
15.5
95.3
22.9
93.0
10
33.8
21.6
36,0
30,8
14.0
35.5
22.7
38.6
MINE-based
20
41.9
25.0
49.5
25.4
22.2
37.5
33.2
39.0
37.7
27.6
64.3
27.8
33.1
22.6
37.4
Cycle loss
20
$2.0
23.6
58.5
29.2
18.2
35.9
23.4
38.8
40
53.8
20.7
62.8
22.4
19.3
31.6
24.5
35.8
Ø-VAE
24.9
27.0
25.6
30.5
EC VAE (c)
95.1
57.7
22.6
EC*-VAE & s)
95.2
13.6
95.1
19.6
25.2
96.2
31.0
91.0
4 EXPERIMENTS
We evaluate V3 on both synthetic and real data to evaluate its effectiveness and generalizability in different domains and scenarios, covering audio, image and video data. The highlight of this section is that V3 effectively learns disentangled representations of content and style, performs well on out-of-distribtion generalization, and the discrete content representations manifest symbolic- level interpretability that aligns well with human knowledge. We also provide additional results in Appendix C and the ablation study in Appendix D
We compare V3 with three unsupervised baselines: 1) an unsupervised content-style disentanglement based on MINE (Tjanra et al. (2020a), and 2) a 2-branch autoencoder similar to our architecture choice, but trained with the cvcle consistency loss after decoding and encoding shuffled combinations of nt and 8 (Zhu et al. 2017). . 3) vanilla 3- (Higgins et al., 2017). Additionally, we compare with two methods with label supervision: 1) weakly-supervised method for disentanglement named ECC, in which the model is trained to predict the correct content labels from zf, as a replacement of the VQ layer, and the decoder is trained to reconstruct inputs from 2; and ground truth content labels (Yang al., 2019 Wang al. 2020 and 2) fully supervised variant EC VAE provided with both content and style labels, in which the model learns to predict both content and style from their latent representations. We denote them as EC*-VAE (c) and EC2-VAE (c s) respectively. All reported results are the average of three best-performing checkpoints on validation sets. We provide further details model architectures in Appendix
4.1 DATASETS
Written Phone Numbers Dataset (PhoneNums): We synthesize an image dataset of written digit strings on light backgrounds using different ink colors, mimicking scenario of handwritten phone numbers. The order of digits is random. All images are diversified with noises, blur, and foreground and background color jitters. Models should learn digits and colors as content and style.
Monophonic Instrument Notes Dataset (InsNotes): We synthesize a dataset consisting of 16kHz monophonic music audio of different instruments playing 12 different pitches in an octave. Every pitch is played for second with random velocity and amplitude envelope. The audio files are then normalized and processed to magnitude spectrograms. Models should learn pitches and timbres content and style, respectively.
Street View House Numbers (SVHN) Netzer et al. 2011): We select all images with more than one digit from the SVHN dataset. We crop the images the bounding boxes of the digits and resize them to 32x Models should learn digits as content, their fonts, texture and colors as style. Note that the styles can be seen from continuous space, and the fonts in SVHN very diverse.

---

Published as a conference paper at ICLR 2025
Table Evaluation of pitch and timbre disentanglement on InsNotes using latent retrieval. Values are reported in percentage.
Content
Style
PR-AUC
Best Fl
PR-AUC
Best Fl
Method
2"
2"
2"
2"
12
89.9
8.9
90.1
15.1
9.3
87.5
15.0
88.0
76.2
8.7
80.0
14.2
12.8
68.9
20.3
70.0
48
72.2
8.4
74.4
14.2
12.3
72.2
22.0
71.5
12
56.4
7.61
62.0
14.2
10.3
61.4
16.9
63.7
MINE-based
24
50.5
8.5
59.1
14,9
14.7
53.4
19.5
51.4
18.3
57.9
15.2
18.2
Cycle loss
24
47.0
8.7
54,5
15.2
14.2
18.9
19.4
23,1
48
42.4
8.0
49.4
14.5
16.2
20.0
22.4
24.4
Ø-VAE
18.1
20.8
12.2
19.0
EC VAE (c)
14.2
EC* & s)
90.4
90.4
14.2
11.1
90.5
18.0
90.4
Sprites with Actions Dataset (Sprites) (ope Yingzhen and Mandi 2018): The Sprites dataset contains animated cartoon characters with random appearances. We use modified version of the dataset taking video sequences of characters performing different actions in random order. Models should learn the actions as content and appearances as style. Note that the styles can be seen as from a continuous space.
Librispeech Clean 100 Hours (Libri100) (Panayotov et al. 2015): Librispeech is large-scale multi-speaker corpus read English speech in various accents. We use the "clean" pool of the Librispeech dataset, where we select the 100-hour subset for training. We align the audio to 39 phonemes (24 consonants and 15 vowels) using ground truth transcriptions with the Montreal Forced Aligner (McAuliffe al. (017), then extract 80-dimensional log-mel spectrograms and resize them to length of 64 frames. Models should learn phonemes as content and speakers' voice as style. The styles considered from continuous space as the speakers' voices are very diverse.
4.2 RESULTS OF CONTENT-STYLE DISENTANGLEMENT
On PhoneNums and InsNotes where concrete style labels are available, we evaluate the models' content-style disentanglement ability by conducting a retrieval experiment to examine the nearest neighbors of every input z" and z" using ground truth content and style labels, evaluated by the area under the precision-recall curve (PR-AUC) and the best Fl score. We experiment with different codebook sizes K to allow different levels of vocabulary redundancy. The results shown in Table D and Table 2 We see that V3 outperforms unsupervised baselines on both datasets, and the performance is consistent across different codebook sizes K. V3 also outperforms EC2-VAE (c) in the style retrieval task, which indicates that V3 learns netter-disentangled style representations containing less content information. Visualizations of content and style latent representations learned by V3 also show clearer grouping compared to baselines, for which we refer readers to Appendix C.I
Table Linear probing accuracies (in %) for content (digit) classification on SVHN.
Table Linear probing accuracies (in %) for content (action) classification on Sprites.
Method
K
Method
K
1
18.5
V3
88.2
20.2
MINE-based
36.0
20.8
MINE-based
79.
22.2
Cycle loss
16.8
21.2
Cycle loss
86.4
39.7
3-E
21.8
3-VAE
33.2
Raw input
21.4
Raw input
99.0
EC VAE (e)
97.0
21.2
BCR VAE (c)
99.8
15.7

---

Published as a conference paper at ICLR 2025
Table 5: Linear probing accuracies (in %) for content (phoneme) classification on Libri100.
Table 6: Speaker verification equal error rates (in %) with average embedding on Libri 100.
Method
K
Method
K
2" 1
V3
80
52.1
40.4
V3
80
49.5
42.5
MINE-based
SU
28.6
51.6
MINE-based
49,9
45.2
Cycle loss
80
50.5
Cycle loss
49.8
45.9
5-VAE
11.0
5-VAE
50.0
Raw imput
31.8
Raw input
47.1
BCC.VAE (c)
78.1
18.2
EC (c)
49.2
49.6
On SVHN and Sprites where there are style labels, we evaluate the models' disentanglement ability by linear probing on the learned representations to predict content labels. We also compare with a linear classifier
raw input features. The classifier layer is trained for one epoch before evaluated on the test set. On both datasets we allow a 100% content vocabulary redundancy, resulting in K 20 for SVHN and K 18 for Sprites. The resulting accuracies shown in Table 3 and Table V3 outperforms unsupervised baselines on both datasets, only trailing behind the weakly supervised EC VVE (c) the latter's z" space is optimized for discriminative task.
On Libri100, we evaluate the disentanglement ability by linear probing on the learned representations predict content labels, well conducting vanilla speaker verification experiment using the average embeddings of fragments in every utterance. We also allow a content vocabulary redundancy of about 100% (K 80). The results are shown in Table 5 and Table 6 The content and style embeddings learned by V3 shows better performance on their respective tasks and lower performance on the other task, indicating that V3 learns better disentangled representations.
4.3 CONTENT CLASSIFICATION ON DINF-DIIISTIIIOON STYLES
We further evaluate the generalization ability of V3 on PhoneNums and InsNotes by testing the models' content classification performance on a special test set with only unseen styles, provided with few-shot examples. We focus comparing V3 with the weakly supervised method EC (c) and a pure CNN classifier to evaluate the generalization ability introduced by latent disentanglement, In the n--hoot settings, models are presented with n samples of each content and new style combination. All models are continuously trained on new samples until performance stops improving. For V3, we choose the V3 versions with no codebook redundancy for comparison (K 10 for PhoneNums, K K=122 for InsNotes) as they show a one-to-one mapping from codebook entries to content labels (see Section 4.4 and Appendix C.2 for details). We first align the learned codebook entries to ground truth content labels, and obtain classification results by the encoded content representations zº. For ECR-VAE, we try two different continuous training strategies: 1) using pseudo content labels from its own predictions for self-boosting as well as training the reconstruction loss, and 2) only optimize the reconstruction loss. Additionally, we compare with 2 VAE and the CNN classifier provided with labels in continuous training. The results are shown in Table 7 Although V3 might fall behind supervised methods in the 0-shot setting, it comes to the lead in few-shot settings on both datasets as the number of extra samples increases. This indicates that V3 learn by itself to make sense of unseen styles with only few examples, an ability that emerges from learning representations and disentangied interpretable factors.
Table Content classification accuracies (in %) on data with OOD styles.
Pretraining
Continuous Training
PhoneNums
InsNotes
Method
Supervision Supervision Self-boost 0-shot
I-shot 5-shot 10-shot
(Ishoo
I-shot 5-shot 10-shot
V3
No
57.8
91.3
97.1
99.0
90.5
97.6
97.8
99.2
EC3-VAE (c)
Yes
No
84.2
92.1
92.2
92.7
87.1
89.4
91.2
-CE (c)
Yes
No
84.2
91.8
92.1
92.4
87.1
94.6
95.0
95.1
CNN Classifier
Yes
No
No
59.5
59,5
59.5
59.5
92.6
92.6
92.6
92.6
CNN Classifier
Yes
No
Yes
59.5
80.2
82.2
82.7
92.6
87.6
85.9
85.3
EC VAE (c)
Yes
Yes
84.2
94.6
98.8
99.2
87.1
97.7
98.9
CNN Classifier
Yes
Yes
No
59.5
81.2
82.4
83.5
92.6
91.9
91.3
89.1
8

---

Published as a conference paper at ICLR 2025
Table 8: Quantitative results of codebook interpretability on datasets with discrete style labels. Values are reported in percentage.
PhoneNums
InsNotes
Method
K
Acc.
0
Acc. f
10
89.2
0.6
12
99.8
0.1
V3
20
99.7
1.8
24
92.9
2.2
40
99.9
4,5
48
90.2
4.5
10
40.9
13.8
MINE-based
20
25.6
29.4
40
50.6
6.2
48
26.9
3.6
10
71.0
3.7
12
27.5
11.4
Cycle loss
20
89.6
28.5
11.9
40
99.9
18.2
Table 9: Quantitative results of codebook interpretability on datasets without discrete style labels. Values are reported in percentage.
SVHN
Sprites
Libri100
Method
K
Acc. 1
K
Acc.
K
Acc.
V3
47.6
98.5
80
24.7
MINE-bbsed Cycle loss
26.0
38.3
80
10.8
20.1
82.0
80
10.5
4.4 RESULTS OF SYMBOLIC CONTENT INTERPRETABILITY
We notice that interpretable symbols emerge the learned codebook of V3, showing its ability of ab- stracting concepts from information. To evaluate the interpretability learned content representations quantitatively, we propose two metrics: the learned content codebook accuracy and standard deviation among styles. We first align codebook entries the ground truth content labels by their distributions on content labels, and then calculate the accuracy of codebook entries' distribution regarding their aligned labels. well learned interpretable codebook should have entries concentrated on content labels they are aligned with, thus showing high accuracy. Also, as good symbol is symbol of consensus, on datasets with discrete style labels, we also quantify different styles' discrepancy of codebook entries distribution between codebook entries and content labels, as shown in Table S For datasets with no discrete style labels, report the accuracy of codebook entries in Table Visualizations of confusion matrices between the codebook and ground truth content labels can be found Appendix C222
content labels using the standard deviation (σ) of confusion matrices
From Table and Table 9 observe that V3 shows good codebook interpretability by representing content labels with consistent codebook entries, and the consistency is kept well among styles. Table also shows that V3 shows good codebook interpretability with without vocabulary redundancy, indicating V3 does not rely on the knowledge of the number of content classes to learn interpretable symbols.
Qualitatively, we perform content and style recombination by traversing all content codebook entries and decoding them with fixed style representation. If the learned codebook has good interpretability. the decoding results of recombined content and style representations should show meaningful content changes, and retain consistent styles. Here, we focus on comparing the recombination results of V3 and baselines on SVHN, where we first encode 2 from example fragments, and then recombine it with all content codebook entries. More experiment results on PhoneNums and InsNotes can be found in Appendix C.2 and our demo website
From Figure we observe that V3 generates images with clear content changes and consistent styles when recombining content and style representations. Although images generated by V3 may not cover all possible contents as many of them never appear in the training set the given styles, the content almost all recognizable digits and V3 can even "imagine" reasonably what digit would look like a new font and color. In contrast, the baselines generate images with either mixed content and style information, very subtle changes in content that are hard to interpret. This comparison

---

Published as a conference paper at ICLR 2025
source
2F
V3
MINE-based
Cycle loss
10110225566221111777
101225516321551671
16601122111144555000000
2198:1122227/85222211
722216122222222225
22222222222222
101732554632211512222
0333888883333333533
1011325331
63651435551455366666
655522215552255222
5555555555555555
(616666666246617775
669996666
666666666666666666
Figure 3: Comparison of generated images by recombining z" from given sources in SVHN and all z" in the learned codebook.
not only validates the interpretabilityy of V3's learned codebook resulted from successful content-style disentanglement, but also demonstrates the potential of V3 in style transfer and content editing tasks.
LIMITATION
We have identified several limitations in our V3 method that necessitate further investigation. First, while V3 achieves good disentanglement and symbolic interpretability, it is not flawless - samples of different contents (say images of "8" and "9") may be projected into the same latent code. Inspired by human learning, which effectively integrates both mode- and mode-2 cogmitive processes, we aim to enhance V3 by incorporating certain feedback or reinforcement. This adaptation could also facilitate the application of V3 more complex domains such as general image or video. Also, V3 focuses on learning the atomic content symbols that compose meanings of data samples, the current approach does not learn the meanings of individual symbols those emerging from specific symbol permutations. Additionally, V3 is currently optimized to disentangle content and style from data samples that include defined fragments. Extending this capability to unsegmented data of large vocabularies, such as continuous audio, represents a significant area for future development. Furthermore, V3 assumes that content elements do overlap, which does not hold in cases of polyphonic music or mixed audio. Addressing this challenge will require a more sophisticated approach that considers the hierarchical nature of content.
CONCLUSION
In conclusion, we contributed an unsupervised content-style disentanglement method named V3, V3's inductive bias is domain-genent intuitive, and concise, solely based on the meta-level insight of the statistical difference between content and style, i.e., their distinct pariance-invariane patterns reflected both within and across data samples. Experiment results showed that V3 not only outperforms the baselines terms of content-style disentanglement, but also demonstrates superior generalizability on OOD styles compared to supervised methods, and achieves high interpretability of learned content symbols. The effectiveness V3 generalizes across different domains, including audio, image, and video. We believe that V3 has the potential to be applied to emergent knowledge in general, and plan to extend our method to more complex tasks and domains in the future.
REFERENCES
About the Liberated Pixel Cup lpc.oppeggamart..rgg. http:::/llpc.opeeggmeeeeee [Accessed 26-09-2024].
Junwen Bai, Weiran Wang, and Carla Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 34: 10100-101111 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invarte iaaama---ivaiaanc-ccvvaaaaaaa regularization for self-supe learning. In 10th International Conference on Learning Representations, ICLR 2022, 2022.
10

---

Published as a conference paper at ICLR 2025
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning. pages 531-540. PMLR, 2018.
Russell Sammut Bonnici, Martin Benning, and Charalampos Saitis. Timbre transfer with variational auto encoding and cycle-consistent adversarial networks. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2022.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 3722-3731, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789-8797, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 8188-8199 1977, 2020.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint arXiv:2307.04686 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 2017.
Ziming Hong, Zhenyi Wang, Li Shen, Yu Yao, Zhuo Huang, Shiming Chen, Chuanwu Yang, Mingming Gong, and Tongliang Liu. Improving non-transferable representation learning by harnessing content and style. In The Twelfth International Conference on Learning Representations, 2023.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems, 30, 2017.
Phillip Isola, Jun-Yan Zhu Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 1125-1134, 2017.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9865-9877 2019.
Hirokazu Kameoka. Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks. In 2018 IEEE Spoken Language Technology Workshop (SL.T), pages (666-773.7 IEEE, 2018.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve2: Rethinking conditional methods for stargan-based voice conversion. arXiv preprint arXiv:1907, /2279, 2019.
11

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4401-4410, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:/312.61 14, 2013.
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18062-18071, 2022.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1), 2022.
Yinghao Aaron Li, Ali Zare, and Nima Mesgarani. Starganv2-ve: A diverse, unsupervised, non- parallel framework for natural-sounding voice conversion. arXiv preprint arXiv:2107.00394 2021.
Liwei Lin, Qiuqiang Kong, Junyan Jiang, and Gus Xia. A unified model for zero-shot music source separation, transcription and synthesis. arXiv preprint 1.XXi:2200.034556 2021.
Liwei Lin, Gus Xia, Junyan Jiang, and Yixiao Zhang. Content-based controls for music large language modeling. arXiv preprint 0rXiv:2310.17100 2023.
Ming-Yo Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-iman translation networks. Advances in neural information processing systems, 30, 2017.
Xuanjie Liu, Daniel Chin, Yichen Huang, and Gus Xia. Learning interpretable low-dimensional representation via physical symmetry. arXiv preprint arXiv:2302. 10890, 2023.
Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell. Multi-task adversarial network for disentan- gled feature learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3743-3751, 2018
Chien Yu Lu, Min-Xin Xin Xue, Chia-Che Chang, Che-Rung Lee, and Li Su. Play as you like: Timbre- enhanced multi-modal music style transfer. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 1061-1068, 2019.
Yin-Jyun Luo, Sebastian Ewert, and Simon Dixon. Towards robust unsupervised disentanglement of sequential data-a case study using music audio. arXiv preprint arXiv:2205.05877 2022.
Yin-Jyun Luo, Sebastian Ewert, and Simon Dixon. Unsupervised pitch-timbre disentanglement of musical instruments using a jacobian disentangled sequential autoencoder. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1036-1040. IEEE, 2024.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498-502, 2017.
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys, 55(13s):1-42, 2023.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEF, 2015.
12

---

Published as a conference paper at ICLR 2025
Taesung Park, Alexei A Efros, Richard Zhang, and Jan-Yan Zhu. Contrastive learning for unpaired -image translation. In Computer Vision-Eoe EEnn-ECCV// 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part IX 16, pages 319-345, Springer, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen- Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2085-2094, 2021.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171- 5180. PMLR, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 5210-5219. PMLR, 2019.
Kaizhi Qian, Zeyu Jin, Mark Hasegawa-Johnson, and Gautham J Mysore. FO-consistent many-to- many non-parallel voice conversion via conditional autoencoder. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6284-6288. IEEE, 2020a.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-lohnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836-7846. PMLR, 2020b.
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Rethinking content and style: exploring bias for unsupervised disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1823-1832, 2021.
Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409 2017.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv:1611..0 .22000 2016.
Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via masked token prediction and contrastive learning. arXiv preprint arXiv:2106. / /250, 2021.
Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. Unsupervised learning of disentangled speech content and style representation. arXiv preprint arXiv:2010. /2973, 2020a.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Transformer vq-vae for unsupervised unit discovery and speech synthesis: Zerospeech 2020 challenge. arXiv preprint arXiv:2005. /676, 2020b.
Mohammadhassan Vali and Tom Bäckström. Interpretable latent space using space-filling curves for phonetic analysis in voice conversion. In Proceedings of Interspeech Conference, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Proceedings of 21st International Conference on Music Information Retrieval (ISMIR), 2020.
Ziyu Wang, Dejing Xu, Gus Xia, and Ying Shan. Audiotto-symbolic arrangement via cross-modal music representation learning. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 181-185. IEEE, 2022.
Richard Wilhelm, Cary F Baynes, and Carl G Jung. / Ching: Book of changes. Grange Books, 2001.
13

---

Published as a conference paper at ICLR 2025
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Seng-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-auue pretraining with feature fusion and seyword-to-captinn augmentation. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Shaoan Xie, Qirong Ho, and Kun Zhang. Unsupervised image-to-image translation with density changing regularization. Advances in Neural Information Processing Systems, 35:28545-28558, 2022a.
Shaoan Xie, Lingjing Kong, Mingming Gong, and Kun Zhang. Multi-domain image generation and translation with identifiability guarantees. The Eleventh International Conference on Learning Representations, 2022b.
Yangchen Xie, Xinyuan Chen, Li Sun, and Yue Lu. Dg-font: Deformable generative networks for unsupervised font generation. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 5130-5140, 2021.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104, 10157, 20157,002)) 2021.
Ruihan Yang, Dingsu Wang. Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music anal- ogy via latent representation disentanglement. In Proceedings of 230st International Conference Music Information Retrieval (ISMIR), 2019.
Dacheng Yin, Xuanchi Ren, Chong Luo, Yuwang Wang, Zhiwei Xiong, and Wenjun Zeng. Re- triever: Learning content-style representation as token-level bipartite graph. arXiv preprint arXiv:2202.12307 2022.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5679, PMIR, 2018.
Huan Zhang and Simon Dixon. Disentangling the horowitz factor: Learning content and style from expressive piano performance. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Yixiao Zhang, Yukara Ikemiya, Gus Xia, Naoki Murata, Marco Martínez, Wei-Hsiang Liao, Yuki Mitsufuji, and Simon Dixon. Musiemagus: Zero-shot text-to-must editing via diffusion models. arXiv preprint 4rX.v22402.0677. 2024.
Yu Zhang, Peter Tiño, Ales Leonardis, and Ke Tang. A survey on neural network interpretability. IEEE Transactions Emerging Topics in Computational Intelligence, 5(5):726-742, 2021.
Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
14

---

Published as a conference paper at ICLR 2025
APPENDICES
The appendix is structured into 5 main parts. Appendix A provides specifics about the datasets involved in the paper. Appendix presents presents implementation and training details of V3 and baseline methods. Appendix C provides additional experiment results and especially visualizations for better understanding. Appendix DIpresents ablation study the V3 model. Finally, provide analysis on learning content and style in Appendix
A DATASET DETAILS
A.1 PHONENUMS
The written phone numbers dataset is designed to represent clear content and style separation to human. We use the Kristen ITC font for the style because its digits look similar handwritten digits and are easy to distinguish. We render the digits from 0 to 9 on light background of RGB (10, 10, 10) using the foreground colors listed Table 10 For more randomness, we first jitter the foreground
Table 10: List of colors and their corresponding RGB values. Colors only used for out-of-distribution
experiments in Section are marked with
RGB Values #
Color
(8, 8, 8)
Black
(8, 8, 248)
Blue
(8, 128, 8)
Green
(248, 8,
Red
(8, 128, 128)
Teal
(128, 8, 128)
Purple
(248, 163,
Orange
(163, 47, 47)
Brown
(248, 188, 199)
Pink
(243, 128, 115)
Salmon *
(248, 210, 8)
Gold
(8, 248, 8)
Lime
(8, 248, 248)
Cyan
(248, 8, 248)
Magenta
(128, 128, 128)
Gray
(200, 133, 67)
Peru
and background colors by a noise from -2 to along every channel, then add small Gaussian noise. then translate all digits vertically or horizontally by a random number of pixels between -2 and Lastly, we add random Gaussian blur effect. Our dataset contains 100000 images in total, each of which has 10 digits. The dataset is split into the train set, validation set, and test set with a ratio of 8:1:1. Some samples of the dataset can be viewed in Figure 4
2950683147 7340295168
9075681423
4928705316
0781639452
7358124096
8356419027
3917084265
2865714933 8312045769
6824570931
1279548603
37 3749506128 3752618049
5283164790
0687295341
Figure 4: Left: example training data in PhoneNums. Right: example data for out-of-distribune evaluation in PhoneNums.
A.2 INSNOTES
In InsNotes, we collect monophonic music audio played by different instruments. This dataset also designed based on the understanding that this domain exhibits content and style concepts that
15

---

Published as a conference paper at ICLR 2025
clear to human music pitches and instrument timbres. The dataset consists of monophonic music audio rendered from 12 instruments playing 12 different pitches in an octave, which corresponds to MIDI numbers from 60 to 71. All the instruments selected have little exponential decays, so their timbres can be well represented with short audio samples. In the aut-o--distribution generalization experiment in Section 4.3 select four unseen instruments: two with slight exponential decays and two with strong attacks and distinct exponential decays. making the task particularly challenging. We list the instruments involved as well the specific MIDI program selected in Table TT
Table II: List of instruments and their corresponding MIDI program numbers. Instruments only used for out-of-distribution generalization are marked with *
Program #
Instrument
19
Pipe organ
21
Accordion
22
Harmonica
41
Viola
52
Choir aahs
56
Trumpet
59
Muted trumpet
64
Soprano sax
68
Oboe
71
Clarinet
72
Piccolo
75
Pan flute
0
Grand piano *
4
Tine electric piano
73
Flute &
78
Irish flute
For every instrument, we play every pitch for one second one by one with a random velocity between 80 and 120 until every pitch is played 10 times. We synthesize 100 such takes at 16kHz using soundfont library for each instrument and further diversify every note by adding a random amplitude envelope each note. The added amplitude envelope is either linear curve starting and ending at a random amplitude factor between 0.8 and 1.2. The audio files are then normalized and processed with short-time Fourier transform (STFT) with the FFT size of 1024 and hop size of 512 to obtain the magnitude spectrograms, which results in a 512 x 32 matrix for each note. To avoid possible overlap between adjacent notes, we add 0.056 second pause in between, resulting in one transition frame in the spectrogram. Our dataset contains 1200 audio files in total, each of which has 120 notes. The dataset split into the train set, validation set, and test set with ratio of 8:1:1.
sinusoidal curve,
A.3 SVHN
The Street View House Numbers (SVHN) dataset is a real-world dataset that contains images of house numbers collected from Google Street View, and digit-level bounding boxes (Netzer al.) 2011). Examples of the dataset can be viewed in Figure The dataset originally consists of 73257 digits for training, 26032 digits for testing, and 531 131 additional, somewhat less difficult samples, as the extra partition. We split the extra partition into additional training, validation and testing sets with a ratio of 8:1:1. For the content-style disentanglement task, we select all images with at least two labeled digits. and resize the bounding boxes to 32 x 48 pixels. The dataset is preprocessed by normalizing the pixel values to the range of [O, 1]. Compared to PhoneNums, although both being image datasets with digits as content, SVHN is significantly more challenging for the following reasons: 1) The digits in SVHN can be very blurry compared to that in PhoneNums: 2) The digits SVHN come with more flexible styles in totally continuous space, involving different fonts, thicknesses, inclinations, colors, and so on; 3) In every SVHN image, the style variation among digits can be more significant than that in PhoneNums, as there be environmental factors like shadows; 4) The bounding boxes are not always tight, clean and complete, and the digits are not always centered in the image; 6) The classes are very imbalanced. Almost all images come with O, 1 or 2 but very few of them have 8
16

---

Published as a conference paper at ICLR 2025
6
31
16
8
9
58
36
2 2
1p
VG
5
191
801
Figure 5: Example data in the original SVHN dataset. The digits the images are bounded by the red boxes.
9; 5) Most importantly, house numbers are generally very short strings. Among images with least two digits, 57.6% of them have exactly two digits, which means for most styles, there is not full coverage of all digits, and during training, V3 only has a highly incomplete view of the full content vocabulary. We choose SVHN to demonstrate the robustness of V3 in learning content and style disentanglement much more challenging setting.
A.4
SPRITES
The original Sprites dataset, collected from ooe oopeaad and adopted by Yingzhen and Mandi (018)) contains animated cartoon characters the pixel graphic style with random appearances and actions. The original Sprites dataset contains animations of six different actions in four perspectives. We collect the Sprites with Actions dataset used in this study by selecting 3 distinet actions 3 perspectives, resulting in 9 different actions in total, and rendering videos of characters performing actions from these 9 categories randomly, using the critical frames from each action animation. The dataset contains 2 2660 videos in total, each of which has 9 frames. The characters differ in their hair, body, top and bottom, forming 2160 unique characters in total. We use 80% of the characters for training and the rest for validation and testing. Examples of the dataset can be viewed in the right of Figure
A.5 LIBRI100
The Libri100 dataset is a subset of the LibriSpeech dataset (Panayotov et al. 2015). containing the "train-clean- 100", "dev-clean", and "test-clean" subsets. There are 331 different speakers in total, in which 165 are female and 166 are male. There are no overlapping speakers between the train, validation, and test divisions. Given audio files and ground truth transcriptions, with the 39 phonemes used in English using the Montreal Forced Aligner (McAuliffe et al. (017). After normalizing the cropped fragments, we extract the mel spectrograms with a window size of 16ms, hop size of 5ms, and 80 mel bands. The 39 phonemes are indexed as shown in Table 12
align the audio
B IMPLEMENTATION DETAILS
B. MODEL ARCHITECTURE
On InsNotes, we instantiate V3 model using a ResNet18 encoder and a ResNet18T decoder with bottlenecks (He et al.) 2016). In the encoder, the number of channels in the first convolutional layer is set to 64, and gradually increase to 512 in the last layer. The first half of the encoder uses a kernel size of and the second half uses kernel size of 5. The decoder is symmetric the encoder. The latent dimension is set to 512. The total number of trainable parameters is 55M.
On PhoneNums and Sprites, half deep as the pitch and timbre learning task. Similarly, the number of channels the first convolutional layer is set to 16, and gradually increase to 256 in the last layer. The encoder uses
instantiate V3 model using ResNet encoder and a ResNetT decoder
17

---

Published as a conference paper at ICLR 2025
Table 12: List of phonemes with their indices.
Index
Phoneme
Index
Phoneme
Index
Phoneme
0
ch
13
ao
26
1
14
сy
27
2
s
15
hh
28
m
3
16
y
29
ch
4
aw
17
30
5
оy
18
31
6
dx
19
32
W
7
dh
20
33
ac
8
uh
21
ah
34
iy
9
aa
22
er
35
10
d
23
ow
36
11
P
24
sh
37
ih
12
25
b
38
jh
kernel size of 5. The decoder symmetrie to the encoder. The latent dimension is set 512. The total number of trainable parameters is 20M on PhoneNums and 25M on Sprites.
On SVHN, add one more ResNet layer in every ResBlock on top of the ResNet encoder used in PhoneNums and Sprites. The number of channels in the first convolutional layer is set to 32, and gradually increase to 512 the last layer. The first half of the encoder uses kernel size and second half uses kernel size of 3. The decoder is symmetric to the encoder. The latent dimension is set to 768. The total number of trainable parameters is 37M.
On Libri 100, we 256. We deepen the encoder with more ResNet blocks in each layer. The total number of trainable parameters 24M.
similar architecture as InsNotes. but with maximum number of channels of
We use the same neural network architecture as V3 for the MINE-based baseline and the cycle loss-based baseline, except that the style branch of the MINE-based method has variational latent layer. For the MINE-based baseline, we use 3-layer multi-layer perceptron with 512 hidden units to estimate the mutual information. For the supervised baselines V (c), replace the VQ layer of the content branch with linear layer projecting to the dimension of prediction logits. Besides, the encoder output of the style branch are mean and log variance vectors instead of representation vectors, which means the style branch is a variational autoencoder (VAE) (Kingma and Welling 2013). For the fully supervised baseline BC2-VAE (c/s), we project the reparameterized style vectors to the dimension of prediction logits.
B.2 TRAINING DETAILS
For all models, we the Adam optimizer with learning rate of 0.001 (Kingma and Ba 2014). The fragment sizes on PhoneNums, InsNotes, SVHN and Sprites are set to 10. 12, and 6, respectively. The relativity is set 15, 15, 5, 10 and 5 on PhoneNums, InsNotes, SVHN, Sprites and Libri 100, respectively (Generally, we recommend setting a higher r. such as 15, on datasets with clean content complex datasets where reconstruction
and style separation, and setting lower r. such 5, might need to be emphasized m. The V3 loss weight 3 is defaultly set to on InsNotes task, and 0.1 other datasets. For all VQ-based models. we update the codebooks using exponential moving average with a decay rate of 0.95 (Van Den Oord et al. (017). The commitment loss weight set to 0.01. On PhoneNums and InsNotes, we set threshold of TOK for dead code relaunching to improve codebook utilization, where n is total the number of fragments in batch. On SVHN, as most images have only content coverage in practice to stabilize training. Similarly, on Librispeech, phonemes do not exhibit distinct styles like vowels, we also smooth the styles by taking the average of adjacent fragments in practice. For MINE-based baseline models, update the MINE network once every global iteration using the Adam optimizer and adaptive gradient scaling Tjandra et al. (2020a): Belghazi al. (2018). The learning rate of the MINE network is set to 0.0002.
digits, we concatenate fragments in different samples for higher
many consonant
18

---

Published as a conference paper at ICLR 2025
We train all models using exponential decay learning rate scheduler, and take the model with the best validation loss as the final model. All models are trained on a single Nvidia RTX 4090 GPU. The V3 loss should decay to zero within few epochs after training starts. All supervised learning methods converge within 2 hours, while the converging time all unsupervised learning methods differs from 5 hours to 24 hours.
MINE-besed
Cyele loss
Style
Purple
Orange
Figure 6: t-SNE visualization of the learned digit (content) and color (style) representations on PhoneNums when there is codebook redundancy (K 10).
MINE-based
Cycle loss
K-20
Content
2
5
8
Style
Black
Green
Teal
Purple
Brown
Figure t-SNE visualization of the learned digit (content) and color (style) representations on PhoneNums when the codebooks are redundant.
V3
MINE-based
Cycle Joss
&O-VAE
Content
F8
B
Style
Soprano Sax
Pipe Organ
Accondion
Trumpet
Muted Trumpet
Obee
Clarinet
Piecolo
Pan Flute
Hammonica
Choir Aahs
Figure 8: t-SNE visualization of the learned pitch (content) and timbre (style) representations on InsNotes when there is codebook redundancy (K 12).
19

---

Published as a conference paper at ICLR 2025
V3
MINE-basee
Cycle loss
K-24
K-48
X--24
K-48
K-24
K-48
Content
Di
AR
Style
Suprano Sax
Pipe Organ
Accordion
Viola
Trumpet
Muted Trumpet
Oboe
Clarinet
Piccolo
Pan Flute
Harmonica
Choir Aahs
Figure 9: t-SNE visualization of the learned pitch (content) and timbre (style) representations
InsNotes when the codebooks are redundant.
C MORE EXPERIMENT RESULTS
In this part, we provide extra experiment results in addition to the results in Section 4 more on the visualizations the learned content and style representations, and the alignment between the learned codebooks and the ground truth content labels.
We will focus
C.1 RESULTS OF CONTENT-STYLE. DISENTANGLEMENT
This section provides 3-dimensional t-SNE visualization results of the learning content and style representations in support of Section 4.2 We show that on different datasets and under different K settings, content and style representations learned by V3 show the clearest groupings compared baselines, and the groupings match well with ground truth content and style labels.
On PhoneNums, first visualize with t-SNE the learned content and style representations when there is no codebook redundancy (K = 10), and color them by the ground truth content or style 12 for learning digits and colors. The results are shown in Figure 6 We can
labels. We that V3 learns clearer content and style representations in groups compared to unsupervised baselines. When the codebooks contain redundancy, the results are shown Figure We can see that V3 still achieves the clearest content and style grouping.
On InsNotes, the visualizations of x" and x" when K 12 are shown in Figure g and the visualiza- tions when codebook is redundant are shown in Figure 9 Both results also show V3 groups content and style better than baselines.
On SVHN, the visualizations of 2" are shown in Figure 10 Since SVHN does not have discrete style labels, we only show the grouping of content representations. V3 is trained K 20. Although not as good as the results shown in Figure on PhoneNums, V3 still achieves the best grouping of digits with learned content representations among unsupervised methods.
V3
MINE-based
Cycle loss
Figure 10: t-SNE visualization of the learned content (digit) representations SVHN.
20

---

Published as a conference paper at ICLR 2025
On Sprites, there is no discrete style label either. Figure shows the t-SNE visualizations of x", and V3 is trained at K = 18. Both V3 and Cycle loss achieve good content grouping, but it is observable that some clusters of the cycle loss z" have broken into several subclusters, indicating that there is still content and style entanglement. This also supported by Section 4.2
V3
MINE-based
Cycle loss
Cunsent
Figure 11: t-SNE visualization of the learned content (action) representations on Sprites. The 10 colors refer to 10 different actions.
C.2 RESULTS OF SYMBOLIC CONTENT INTERPRETABILIT
This section provides intuitive visualizations about how the learned content codebook entries align with ground truth content labels in Section 4.4 We first collect frequencies of every content encoded to every codebook entry, and then permute the codebook to make the confusion matrix look like an eye for a clear alignment. Then we plot heatmaps of confusion matrices between codebook entries (vertical axes) and content labels (horizontal axes).
On PhoneNums and InsNotes, we plot the confusion matrices under different K settings in Figure 12 and Figure [3] respectively. The results show V3 achieves the clearest symbol interpretability in all K settings. Results on SVHN and Sprites are shown in Figure [4] and Figure T5 On Sprites, both and cycle loss learns codebooks with good interpretability, but V3 still has fewer misclassifications. Although V3 does not learn a clear on-to-one codebook entry to content label mapping on SVHN, still shows clearer alignment relationship than other methods. An interesting fact is the order of learning we observe during training - V3 usually first distinguish O and 1, then start to understand 2 is different, then 3. It often confuses between 5 and 6 and between 1 and 7, and it usually fails to learn 8 and 9. This human-likk learning trajectory might be subject to both the ratio content classes and their pairwise similarities shape. The similar phenomenon is observed in the confusion matrices on Libri100 as shown in Figure 16 V3 distinguishes between consonants and vowels pretty well, and confuses between "z" and "s" and between "n" and "ng", which are phonetically similar.
To further investigate the disentanglement ability of models, we perform latent representation re- combination using the trained models. Figure hass already demonstrated the results on SVHN of decoding fixed z with every z". Here we show the results on PhoneNums, where instead of using a fixed 2 encoded from an example, we compute the mean of all fragments from a class as its style representations for decoding. We select the V3 model with K 10, align codebook entries with digit labels, and enumerate all combinations = and 2², We present the results Figure 17 Compared to baselines, V3 can fairly well reconstruct the involved 8 colors and the digits from 0 to 9, even though it not informed with any discrete labels during training. In contrast, the MINE-baser baseline and the cycle loss baseline fail to distinguish the digits, although the color reconstruction is not bad. They generate blurry digits that look like "5", "8" or "6", which are the most conservative choices. As for results the music dataset InsNotes, refer you our web demo page for interactive experience.
D ABLATION STUDY
For ablation, we experiment with another type of variability measurement vx(:), which standard deviation (SD). Besides, we train four variants of V3, each without one of the four regularization terms defined in Equation 9 12 We conduct experiments on PhoneNums and InsNotes, two datasets with style labels available, and evaluate the content and style disentanglement performances. The results are reported in Table and Table [4] It can be seen that V/3 SD does not work as well as en MPD, which can be explained by its weakness in constraining multi-peak content distributions
21

---

Published as a conference paper at ICLR 2025
V3
MINF-based
Cycle loss
K-10
K-10
K-10
1.0
0.8
K=20
K=40
K=20
K=40
K=20
K=40
Calobook hns
!
0.2
1 5.6
0.0
Figure 12: Confusion matrices learned codebooks on PhoneNums. The horizontal axes show digit labels from "0" to "9", and the vertical axes show codebook atoms sorted by ground truth digit labels.
V3
MINE-based
Cycle loss
K-12
A-12
K-12
1010 Cookbooo Todex
10 Colbboot Brdden
0.8
10
10
Custeot Index
Ourttent Index
Cuaticat Tnclox
X-24
K-48
X-24
X-48
X-24
K-48
0.4
Codebook Index
0.2
Content Index
Content Index
Content Index
Content Index
Content Index
Contcol Index
Figure 13: Confusion matrices of learned codebooks on InsNotes. The horizontal axes show pitch labels from "C" to "B", and the vertical axes show codebook atoms sorted by ground truth pitch labels.
22

---

Published as a conference paper at ICLR 2025
V3
MINE-based
Cycle loss
0.8
Codebook Index
Codebook Index
Codebook Index
0.6
10
14 10
111 10
11
12
12
0.4
11
15 14
37 16 15
0.2
0.0
Content Index
Content Index
Content Index
Figure 14: Confusion matrices of learned codebooks on SVHN. The horizontal axes show digit labels from "0" to "9", and the vertical axes show codebook atoms sorted by ground truth digit labels.
V3
MINE-based
Cycle loss
1.0
0.8
Codebook Index
Codebook Index
14 Codebook Index
0.6
0.4
11
0.2
-..0
Content Index
Content Index
Content Index
Figure 15:
Confusion matrices of learned codebooks on Sprites. The horizontal axes are different action labels, and the vertical axes show codebook atoms sorted by ground truth action labels.
within samples. is also worth noting that V3 sometimes performs fairly well even when discarding one of terms. these cases, we observe a decrease in the discarded loss even if we do not explicitly optimize for it. We suspect this is due to the robustness of V3 constraints reflected in the symmetric relationships among the four losses we can enforce three relations, and the fourth one may fall into the right place automatically. However, in practice, it is difficult to tell the one term to free beforehand as it is also related to detailed content and style variations in specific domains. As result, the V3 constraints whole shows robust performance across domains.
23

---

Published as a conference paper at ICLR 2025
V3
MINE-based
Cycle loss
1.0
2022202
0.8
Codebook Index
Codebook Index
Codebook Index
0.6
0.4
0.2
0.0
Content Index
Content Index
Content Index
Figure 16: Confusion matrices of learned codebooks on Libri100. The horizontal axes are different
phoneme labels, and the vertical axes show codebook atoms sorted by ground truth phoneme labels.
V3
MINE-based
Cycle loss
0
559 5 55955557
3 2
1
11141112
1
1
2
11111
65955555
2 2
2
2
55955557
2 2
3
33333
3 33
5 55955555
3 8
8
444
4. 4
5 8 88988888
8
2 8
4
5
5 5 55955555 55555555
2
8 8
6
66669666
66666666 6
3
8 8
7
7 7777777
6 6 66666668
6
8
8 8
8
8888 888
5 5 55955559
0
2
8
9
9
9.
9 9
9
9
9
5 8
8
8
8
8
8
8
2 2
Figure 17: Comparison of generated digits by recombining content and style latents using unsuper- vised methods trained on PhoneNums.
E DISCUSSION
Connection between Content-Style Disentanglement and OOD Generalizability: Disentangle- ment can intuitively boost OOD generalization for several key reasons. By separating different factors, like content and style, the model can focus on the important features without getting distracted by irrelevant variations. This separation makes the model more robust changes. For instance, if the style changes in an OOD sample while the content remains similar, the model might still recognize and process the content effectively. Additionally, disentangled representations often lead more
24

---

Published as a conference paper at ICLR 2025
Table 13: Ablation study V3 settings on content-style disentaggleenn performance on PhoneNums. Values are reported in percentage.
Content
Style
PR-AUC
Best F1
PR-AUC
Best FI
Method
2º
2²
x²a
V3
10
83.2
12.8
84.1
18.5
14.9
95.4
22.6
91.0
V3 (V/- SD)
10
42.7
17.9
53.5
21.5
18.2
49.9
24.7
51.6
V3 (w/o Lventen))
10
43.8
13.8
51.2
18.9
18.1
90.8
22.4
87.5
V (w/o Lstyle)
10
64.6
13.2
70.3
18.7
17.7
87.8
24.5
83.9
V3 (w/o Litagmen)
10
96.3
11.7
98.9
17.9
15.9
90.1
23.2
88.5
V3 (w/o Lsample
10
47.0
12.7
57.8
18.9
15.4
88.4
24.7
85.3
Table 14: Ablation study of V3 settings on content-style disentanglement performance on InsNotes. Values are reported in percentage.
Content
Style
PR-AUC
Best F1
PR-AUC
Best FI
Method
K
f
82a
2º 4
2
2
V3
12
89.9
8.9
90.1
15.1
9.3
87.5
15.0
88.0
V3 (2) SD)
12
12.9
9.9
17.5
15.0
16.3
24.7
24.1
36.5
V3 (w/o Lcoment)
12
19.2
9.3
28.0
14.3
13.6
66.2
19.0
68.4
V3 (w/o Laste)
12
72.1
8.9
14.2
84.0
13.7
78.7
23.6
79.0
V3 (w/o (tragment)
12
26.0
12.1
35.7
17.6
13.5
53.7
20.1
56.7
V3 (w/o Ssample)
12
86.4
7.9
89.3
14.2
11.3
50.7
19.4
56.2
generalized features, enabling the model identify important patterns that are invariant across differ- ent distributions. This property facilitates easier transfer learning because models with disentangled representations can be more readily fine-tuned for new tasks, as supported by our experiments in Section 43
Connection between Content-Style Disentanglement and Symbolic Interpretability: In Sec- tion 4.2 and Section 4.4 we separately examined content-style disentanglement and symbolic- level interpretability. This discussion now seeks to understand how these elements are intercon- nected speccifiill. whether V3's disentangled representation space inherently improves symbolic- level interpretability.
The transition from purely observational data to symbolic representation remains open question in cognitive science and artificial intelligence. We suggest that robust content-style disentanglement closely linked to better symbolic interpretability, as evidenced in Tables Io and These figures show that both V3 and supervised methods, which achieve better disentanglement, also provide superior interpretability compared to methods less effective in disentanglement (supervised classification - while it is not can here be viewed as another form of interpretable VQ symbols). Additionally, as illustrated in Figures 6 [T] [I]. dill-diienttngeed style spaces (meaning they contain less content information) see well-formed clusters, which can facilitate straightforward postprocessing for discrete and symbolic labeling.
The statistical patterns and mutual relationship between content and style closely correspond to the philosophical concept of Yin and Yang. fundamental duality in universal balance and dynamics. In the famous Yin-Yang diagram, Yin and Yang are equally divided and composed of two identical shapes (the fish-like swirl and the small dot) with opposing colors (light and dark), together forming the completeness of the world Wilhelm et al. (2001).
Connection between Content-Style Disentanglement and Philosophy:
We can interpret the two fundamental shapes the two axes along which data is observed the swirl represents observation across samples, while the dot represents observation across fragments within samples. The two colors signify two kinds of dynamics invariant. As result, the duality Yin and Yang becomes the duality of content and style. Content shows variability within samples (the light dot) and invariability of vocabulary across samples (the
light denotes variant, and dark denotes
25

---

Published as a conference paper at ICLR 2025
Variau within sample
Content
Actoss samples
Across frugmenis
Invariant vocabulary samples
Variant sumples
Variant
Invariant
Style
Invoriant among fragments within sumple
Figure 18: An illustration of the correspondence between the content-style duality and the Yin- Lan- duality.
dark swirl), while style shows variability across samples (the light swirl) and invariability within sample (the dark dot). They can be disentangled from data as two components, yet neither can exist alone.
V3 and related works: We explicate the difference and connection between V3 and several other relevant works below.
InfoGAN (Chen et al. 2016): InfoGAN is similar to approach in that both models learn interpretable representations and decouple these representations from the data. However, there several key differences: 1) Each representation in InfoGAN is of very low di- mensionality; 3) The specific aspects learned are less controllable, while V3 focuses on learning the distinctions of content and style; 3) GAN is known to be less unstable in training than autoencoders and VAEs, and it is a framework more for generative modeling than representation learning.
DSAE and variants (Hsu et al. 2017 Yingzhen and Mandi 2018 Bai al. 202 Luo et al. 2022; 2024): DSAE shares similar insights with regarding the intrinsic rel relationship between content and style, but it primarily focuses the invariability of style and the variability of content within a sample, giving less attention to the invariability content vocabulary and the variability of style across broader scope. Other distinctions include: 1) DSAE focuses on learning a fixed style representation for whole sample, which may struggle with samples where the style varies, such as singing performances that feature both chest voice and falsetto, or instrument performances with multiple articulations: 2) The DSAE family requires access the entire sequence when encoding style; 3) Most importantly, the content learned in DSAE is context-dependent while V3 emphasizes on learning more universal content representations.
VICReg (Bardes al., 2022): V3 has similar form of loss function VICReg, and both models leverage variance and invariance among entities to help training representation learning frameworks. However, VICReg focuses on learning single representation for each entity, while V3 focuses on learning disentangled representations. Also, VICReg learns discriminative representations without clear interpretability. but V3 learns interpretable content symbols, and has decoding ability recombine content-style pairs. In fact, we draw on their mathematical representations and the idea of using regularization to prevent latent representation from collapsing, concept also advocated by LeCun (2022).
26


Reviews:
[{"review_id": "uKzrG5lJAq", "reviewer": "Reviewer_GuWP", "summary": "They propose an unsupervised method to disentangle content and style representations.  They use vector-quantized autoencoder architecture and incorporate variance-versus-invariance constraints. Their method benefits from meta-level inductive bias, and shows out-of-distribution generalization. They demonstrate their method’s effectiveness across diverse applications such audio, images and video clips. Moreover, the learned codebook enables interpretable features.", "questions": "- How does the choice of codebook size (K) affect interpretability and disentanglement performance?  \n\n- The method uses specific values for the relativity parameter r and the V3 loss weight beta  depending on the dataset. Could you clarify the intuition behind selecting these values? How sensitive the performance is to different values of these hyperparameters?\n\n- The paper mainly utilizes small datasets with specific content-style boundaries. How does it scale with larger datasets?\n\nMinor issues: \nLine 463: The term \"contennt\" should be corrected to \"content.\"", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "++ Unsupervised approach for separating style and content, which is a plus for domains where labeled data is unavailable or scarce\n\n++ Generalizes across audio, image and video data. \n\n++ Content representations show interpretable properties. \n\n++ Performs well for out of distribution styles", "weaknesses": "-- They assume content elements are distinct, which can limit its application in domains with overlapping content.\n\n-- The method is optimized for data with predefined fragments, which may not generalize well to continuous or unsegmented data types.\n\n-- The experiments are done on relatively small datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4twSJ4LRu", "reviewer": "Reviewer_D5wf", "summary": "This paper introduces an unsupervised method named V3 (variance-versus-invariance) for disentangling content and style representations from a sequence of observations. The method is based on the assumption that content varies significantly within a sample but maintains a consistent vocabulary across samples, while style remains relatively invariant within a sample but varies significantly across samples. V3 integrates this inductive bias into an encoder-decoder architecture and demonstrates its effectiveness across multiple domains and modalities, including music audio, handwritten digit images, and simple animations. The paper reports that V3 outperforms existing unsupervised methods and even surpasses supervised methods in out-of-distribution generalization and few-shot learning. Additionally, V3 exhibits symbolic-level interpretability, aligning machine representations closely with human knowledge.", "questions": "1. It is better to evaluate V3 on  more complex or challenging datasets, such as face datasets FFHQ? And It is better to add discussion on \n potential limitations of V3 when dealing with more complex data containing multiple contents or styles within a single sample.\n2. Since V3 uses different losses to control content and style representations through self-supervised means, is there a risk that the representations intended to learn style might inadvertently capture content instead? How can it be ensured that there is not too much coupling between content and style representations? I suggest to conduct an analysis showing how much content information can be recovered from the style representations and vice versa.\n3. Why is it necessary to apply VQ to content representations to form a codebook? What would be the impact on the learning results if VQ were not used? Seeing corresponding results in the Ablation study could be more convincing. \n4. Are the numbers of content and style representations in V3 the same? Considering that the actual samples may have varying quantities suitable for representing content and style, would setting different numbers for content and style representations in the same training round make a significant difference in the results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The V3 method offers a novel unsupervised disentanglement approach that leverages the universal statistical differences between content and style. This is a creative application of variance-invariance constraints within an encoder-decoder framework, distinct from traditional methods that rely on supervision or paired data.\n2. The loss calculation method proposed by V3, based on content and style, is concise and clear, aligning with human cognitive characteristics. Humans also tend to keep the style of the same sample similar while focusing on the content differences across different samples.\n3. The model proposed in the paper is applicable to the learning of multiple modalities of style and content, including images, music, and animations, demonstrating strong generalizability. Moreover, the loss calculation pattern presented is not limited to the paper model but can also be applied to other related models for style and content learning.\n4. The paper is well-structured, with a clear problem statement, methodology, and experimental evaluation. The use of datasets from multiple domains enhances the robustness of the research findings. The logic is strong, and the hierarchy is clear. The visualization of learned representations also helps readers better understand the results.", "weaknesses": "1. The paper's approach to learning sample content and style is reliant on datasets with a specific structure, which may limit its effectiveness when applied to images with varied or complex styles. For example, the Written Phone Numbers dataset contains only one digit in a specific position on each image, but the style of the digit can vary; in SPRITES, the character's position is fixed, and the form is also relatively uniform. If an image contains more content or more complex content, the model may have difficulties learning content and style.\n2. The Ablation study reveals that certain components of the loss function may adversely affect performance metrics, suggesting a need for a deeper investigation into the roles and interactions of these components within V3.\n3. It appears that V3 may have difficulty distinguishing between some similar contents during recognition. Similarly, although the paper may not mention it, V3 may also have similar problems with distinguishing styles, thus limiting the style discrimination between different samples.\n4.Some important related works[a,b] are missing, which are also rely one some assumption to disentangle content and style.   The detailed comparison should be included, especially on the basic assumptions and the scope of suitable cases. \na. Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph\nb. Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bcynFgrxQ", "reviewer": "Reviewer_oi3G", "summary": "The paper proposes an unsupervised method that learns from raw observation and disentangles its latent space into content and style representations based on the meta-level insight of domain-general statistical differences between content and style.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method does not rely on external supervision such as labels.\n2. The proposed method can generalize to unseen data.\n3. The disentangled content representations align with human perceptions in some cases.", "weaknesses": "1. The paper does not explicitly discuss how the codebook size and codebook dimension are determined. This could limit the application of the proposed method to different datasets.\n2. The experiments are all conducted on toy datasets. This weakens the method's practical use in real-world scenarios. For example, for audio disentanglement, the musical note disentanglement is too simple. It is recommended to perform real-world disentanglement experiments such as voice conversion using VCTK.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jxEcrkqVBz", "reviewer": "Reviewer_ipXQ", "summary": "This paper proposes an unsupervised disentanglement algorithm between styles and content, based on the assumption that style information remains consistent within samples, but varies across samples; whereas content information varies within samples, but the aggregated content is stable across samples due to the fixed vocabulary.", "questions": "Can the author elaborate a bit more on the experiment setting of Section 4.2?\n'we evaluate the models’ content-style disentanglement ability by conducting a retrieval experiment to examine the nearest neighbors of every input zc and zs using ground truth content and style labels, evaluated by the area under the precision-recall curve (PR-AUC) and the best F1 score'", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes an innovative disentanglement algorithm that does not rely on external labels of the style or content, making a novel contribution over the existing work, most of which requires external labels.\n\n2. There is a clear rationale behind the proposed algorithm, and the rationale is interesting\n\n3. The presentation in the paper is very clear.", "weaknesses": "1. Although the proposed method does not rely on any external labels, it seems to be built upon rather strong assumptions. In particular, the method assumes that the aggregated content of a sample should stay constant across different samples, at least more stable than the style. This assumption is very strong, and there are many scenarios where this may not hold. First, if the content information in each example is too small to exhaust all the possible combinations, then the content would still vary greatly across samples. For example, in the written digit case, if each example only contains one digit. Then the model may no longer be very successful. This limitation may also hold for many other scenarios, such as Chinese calligraphy, which typically only contains very few characters, paintings containing only one object, etc. Second, if the content information exhibits too much variability, then the content across different statements may still have more variability than style. For example, consider short passages of different languages, where the content is the meaning of the passage and the style is language. The content in this case still varies greatly across different passages. Meanwhile, the authors also acknowledge that the method cannot deal with large content vocabularies. This limitation, together with the limitation discussed above, significantly limits the applicability of the proposed method.\n\n2. Related to the first limitation, the experiment section seems to focus on very simple, carefully crafted toy scenarios that happen to satisfy the assumptions of the proposed method. There is no experiment on real-world scenarios, such as disentangling real human speech, photos, paintings, music, passages, etc. The lack of real-world experiments adds to the concern that the assumptions made by the proposed method may deviate too much from real-world cases to render the method useful.\n\n3. (Minor) On page 5, the authors state that 'Theoretically, we aim to measure the consistency of codebook usage distribution along the sample axis, which is not differentiable'. This is not necessarily true. We can derive a soft, differentiable distribution of codebook usage via a softmax codebook selection. That is, for each codebook selection, rather than have a one-hot vector indicating which code is selected, we can derive a softmax selection, which is very common in deep-learning-based vector quantization methods, such as VQ-VAE. As a result, I believe that using the mean content vector to represent the distribution information may be improved.\n\n4. (Minor) In line 242, the authors state that 'Content should be more variable within samples than across samples'. This is an overly simple statement. A more precise version should be 'Content should be more variable within samples than the sample-wise aggregated content across samples'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cwbyUtYPtG", "reviewer": "Reviewer_BaC3", "summary": "The authors propose a method to learn representations from an adapted VQ-VAE that disentangle content and style based on their different statistical characteristics: style is more constant over patches of a sample, whereas content can vary.", "questions": "none", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to follow. The method is well motivated and described and results are nicely presented.", "weaknesses": "I have few comments on the main body of the paper, so please do not see this as a \"few line review\", I just do not have much to say.\n\n**Experiments**: \n* 3 of 4 are on synthetic data with only SVHN being a \"natural\" data set and even that is cropped to the task. \n* It is of course right to demonstrate that the model performs well on data that follows its assumptions. \n* However, it would be informative (a) to see results on more natural datasets, and (b) to see how the model performs where assumptions do not hold so well, if only as a benchmark for future improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)