Unsupervised Disentanglement of Content and Style via Variance-Invariance Constraints

## Abstract

Abstract We contribute an unsupervised method that effectively learns from raw observation and disentangles its latent space into content and style representations. Unlike most disentanglement algorithms that rely on domain-specific labels and knowledge, our method is based on the insight of domain-general statistical differences between content and style — content varies more among different fragments within a sample but maintains an invariant vocabulary across data samples, whereas style remains relatively invariant within a sample but exhibits more significant variation across different samples. We integrate such inductive bias into an encoder-decoder architecture and name our method after V3 ( v ariance- v ersus-in v ariance). Experimental results show that V3 generalizes across two distinct domains in different modalities, music audio and images of written digits, successfully learning pitch-timbre and digit-color disentanglements, respectively. Also, the disentanglement robustness significantly outperforms baseline unsupervised methods and is even comparable to supervised counterparts. Furthermore, symbolic-level interpretability emerges in the learned codebook of content, forging a near one-to-one alignment between machine representation and human knowledge . 1 1 1 Demo can be found at https://v3-content-style.github.io/V3-demo/ .

## 1 Introduction

Learning abstract and symbolic representations is an essential part of human intelligence. Even without any label supervision, we humans can abstract rich observations with great variety into a category, and such capability generalizes across different domains and modalities. For example, we can effortlessly perceive a picture of a “cat” captured at any angle or set against any background, we can perceive the symbolic number “8” from an image irrespective of its color or writing style variations, and we can perceive an abstract pitch class “A” from an acoustic signal regardless of its timbre. These symbols form the fundamental vocabulary of our languages—be they natural, mathematical, or musical—and underpin effective and interpretable communication in everyday life.

Our goal is to emulate such abstraction capability using machine learning. We choose a content-style representation disentanglement approach as we believe that representation disentanglement offers a more complete picture of learning symbolic abstractions—concepts that matter more in communication, such as an “8” in a written phone number or a note pitch “A” in a folk song, are usually perceived as content, while the associated variations that often matter less in context, such as the written style of a digit or the singing style of a song, are perceived as style. In addition, content is usually symbolized and associated with rigid labels, as we need precise control over it during communication. E.g., to write “8” as “9” in a phone number or to sing an “A” as “B” in a performance can be a fatal error. In comparison, though style can also be described discretely, such as an “italic” writing or a “tenor” voice, a variation over it is usually much more tolerable.

In the machine-learning literature, significant progress has been made recently in content-style disentanglement for various tasks, including disentangling objects from backgrounds , characters from fonts ; , pitch from timbre ; , and phonemes from speaker identity ; . However, most existing models rely heavily on domain-specific knowledge and require substantial supervision. The supervision forms can be explicit content or style labels ; ; ; ; ; ; ; ; , pre-trained content or style representations ; ; , or paired data showcasing the same content rendered in different styles or vice versa ; . Additionally, the learned representations often lack interpretability at a symbolic level and do not align well with human perceptions ; .

Figure: Figure 1: An illustration of the variance-versus-invariance constraints of content and style.
Refer to caption: x1.png

To address the aforementioned challenges and achieve more interpretable disentanglement in an unsupervised manner, we introduce V3 (variance-versus-invariance), a novel method to disentangle content and style by leveraging meta-level prior knowledge about their inherent statistical differences. As shown in Figure 1, our design principle is based on the observation that content and style display distinct patterns of variation—content undergoes frequent changes within different fragments of a sample yet maintains a consistent vocabulary across data samples, whereas style remains relatively stable within a sample but exhibits more significant variation across different samples.

In this paper, we adopt the vector-quantized autoencoder architecture and incorporate variance-versus-invariance constraints to guide the learning of latent representations that capture style-content distinctions. As for applications, we demonstrate that our method effectively generalizes across two distinct areas: disentangling pitches and timbres from musical data, and disentangling numbers and ink colors from images of handwritten digits. Experimental results show that our approach achieves
more robust content-style disentanglement than baselines and is even comparable to supervised approaches. Moreover, symbolic-level interpretability emerges in a near one-to-one alignment between the vector-quantized codebook and human knowledge, an outcome not yet seen in previous studies. In summary, our contributions are as follows:

- •
Unsupervised content-style disentanglement: We introduce an unsupervised method that successfully learns to disentangle content and style representations without the need for paired data or explicit content and style labels.
- •
Emergence of interpretable symbols: Our innovative training objective fosters the development of interpretable content symbols that closely align with human knowledge.
- •
Generalizability of variance-vs-invariance constraints : V3 is a purely statistical method that relies only on meta-level inductive bias and does not require domain-specific assumptions typically necessary in visual or auditory data processing (such as a relation between foreground and background in the image domain or the relation between f0 and partials in the music domain).

## 2 Related Work

The content-style disentanglement as well as the related style transfer problem has been well explored in computer vision, especially in the context of image-to-image translation. Early works mostly require paired data of the same content with different styles ; , until the introduction of domain transfer networks that can learn style transfer functions without paired data ; ; ; ; ; ; ; ; ; . Although these methods are unsupervised in the sense that they do not require paired data, they still require concrete labels of styles to identify source and target domains, and there are no fully interpretable representations of either content or style.

A similar trajectory of research has also been followed in other domains including speech ; ; ; and music ; ; ; ; ; ; . To mitigate the requirement for supervision, some methods utilize domain-specific knowledge and have achieved better disentanglement results, including X-vectors of speakers ; , the close relation between fundamental frequency and content in audio ; , or pre-defined style or content representations ; ; . Pure unsupervised learning for content and style disentanglement has not been well explored. Notable attempts include mutual information-based methods such as InfoGAN and mutual information neural estimation (MINE) ; ; ; , and low-dimensional representation learning with physical symmetry . But these methods often suffer from the training stability issue or have to follow a low-dimensionality setup.

A technique often associated with learned content is vector quantization (VQ) . Recent efforts have built language models on top of VQ codes for long-term generation, indicating the association between VQ codebook and the underlying information content ; ; ; ; ; ; . A noticeable characteristic of these studies is the use of large codebooks, which limits the interpretability of representations. We borrow the idea of a small codebook size from categorical representations ; , targeting a more concise and unified content code across different styles, while keeping the high-dimensional nature of VQ representations.

## 3 Methodology

Considering a dataset consisting of $N$ data samples, where each sample contains $L$ fragments. We aim to learn each fragment’s content and style representation with the inductive bias illustrated in Figure 1. Intuitively, the fragments within each data sample have a relatively frequently-changing content and a relatively stable style. For different data samples, the style exhibits significant variations and their content more or less keeps a consistent vocabulary. In this paper, we focus on two tasks: 1) learning pitch (content) and timbre (style) representations from audio spectrograms where each sample contains several note fragments, and 2) learning digits (content) and ink colors (style) representations from images of written digit strings where each fragment is a written number.

### 3.1 Model Architecture

The model architecture of V3 is illustrated in Figure 2. Let ${\bm{X}}=\{{\bm{x}}_{ij}\}_{N\times L}$ be the dataset, where ${\bm{x}}_{ij}$ corresponds to the $j$-th fragment of the $i$-th sample. We use an autoencoder architecture to learn the representations of ${\bm{x}}_{ij}$. The encoder encodes the input data ${\bm{x}}_{ij}$ to the latent space, which is split into to ${\bm{z}}_{ij}^{\mathrm{c}}$ and ${\bm{z}}_{ij}^{\mathrm{s}}$. We use vector quantization as the dictionary learning method for content. Every content representation ${\bm{z}}_{ij}^{\mathrm{c}}$ is quantized to the nearest atom in a codebook of size $K$ as $\tilde{{\bm{z}}}_{ij}^{\mathrm{c}}$. The decoder concatenates $\tilde{{\bm{z}}}_{ij}^{\mathrm{c}}$ and ${\bm{z}}_{ij}^{\mathrm{s}}$ and reconstructs the fragment $\hat{{\bm{x}}}_{ij}$. The overall loss function is the weighted sum of three terms:

$$ $\mathcal{L}=\mathcal{L}_{\mathrm{rec}}+\alpha\mathcal{L}_{\mathrm{vq}}+\beta \mathcal{L}_{\mathrm{V3}}\text{.}$ (1) $$

Here, $\mathcal{L}_{\text{rec}}$ is the reconstruction loss of ${\bm{X}}$ and $\mathcal{L}_{\text{vq}}$ is the VQ commit loss :

$$ $\displaystyle\mathcal{L}_{\mathrm{rec}}$ $\displaystyle=\frac{1}{N\times L}\sum_{i=1}^{N}\sum_{j=1}^{L}\|{\bm{x}}_{ij}- \hat{{\bm{x}}}_{ij}\|_{2}\text{,}$ (2) $\displaystyle\mathcal{L}_{\mathrm{vq}}$ $\displaystyle=\frac{1}{N\times L}\sum_{i=1}^{N}\sum_{j=1}^{L}\|{\bm{z}}_{ij}^{ \mathrm{c}}-\mathrm{sg}(\tilde{{\bm{z}}}_{ij}^{\mathrm{c}})\|_{2}\text{,}$ (3) $$

where $\mathrm{sg}(\cdot)$ is the stop gradient operation of the straight-through optimization. The final term $\mathcal{L}_{\mathrm{V3}}$ is the proposed regularization method to ensure unsupervised content-style disentanglement, which we introduce in the rest part of this section. (For more details of the model architecture and data representations, we refer the readers to the appendix.)

Figure: Figure 2: The model architecture of V3. Left: The autoencoder has two branches for content and style respectively, where the content branch has a VQ layer at the encoder output. Right: the V3 constraints, where double-dashed arrows represent measuring the variability by $\nu_{k}(\cdot)$, and solid arrows represent taking the average.
Refer to caption: x2.png

### 3.2 Variability Statistics

We define four statistics to measure the degree of variability in accordance with the four edges of Figure 1. These statistics are based on a backbone variability measurement $\nu_{k}(\cdot)$, where $k$ represents the dimension along which variability is computed.
In this paper, we define $\nu_{k}(\cdot)$ as the mean pairwise distance (MPD). Formally, for a vector ${\bm{z}}$ of length $D$,

$$ $\nu_{i=1}^{D}({\bm{z}}_{i}):=\mathrm{MPD}_{i=1}^{D}({\bm{z}}_{i})=\frac{1}{D(D -1)}\sum_{i=1}^{D}\sum_{j=1,j\neq i}^{D}\|{\bm{z}}_{i}-{\bm{z}}_{j}\|_{2}\text {.}$ (4) $$

The motivation for using MPD is that it is more sensitive to multi-peak distributions than standard deviation, which is preferred when learning diverse content symbols in a sample. We compare different choices of $\nu_{k}(\cdot)$ in Section 4.1.

Content variability within a sample ($\mathcal{V}_{\mathrm{f}}^{\mathrm{c}}$). We first compute the variability of content along the fragment axis and take the average along the sample axis. The value is the average of content codes before and after vector quantization:

$$ $\mathcal{V}_{\mathrm{f}}^{\mathrm{c}}=\frac{1}{2N}\sum_{i=1}^{N}\nu_{j=1}^{L}( {\bm{z}}^{\mathrm{c}}_{ij})+\frac{1}{2N}\sum_{i=1}^{N}\nu_{j=1}^{L}(\tilde{{ \bm{z}}}^{\mathrm{c}}_{ij})\text{.}$ (5) $$

Content variability across samples ($\mathcal{V}_{\mathrm{s}}^{\mathrm{c}}$). Theoretically, we aim to measure the consistency of codebook usage distribution along the sample axis, which is not differentiable. In practice, we compute the center of the content code along the fragment axis and measure the variability of the centers along the sample axis. It serves as a proxy of codebook utilization. Also, we consider both the original content codes (before and after vector quantization):

$$ $\mathcal{V}_{\mathrm{s}}^{\mathrm{c}}=\frac{1}{2}\nu_{i=1}^{N}\bigl{(}\frac{1} {L}\sum_{j=1}^{L}{\bm{z}}^{\mathrm{c}}_{ij}\bigr{)}+\frac{1}{2}\nu_{i=1}^{N} \bigl{(}\frac{1}{L}\sum_{j=1}^{L}\tilde{{\bm{z}}}^{\mathrm{c}}_{ij}\bigr{)} \text{.}$ (6) $$

Style variability within a sample ($\mathcal{V}_{\mathrm{f}}^{\mathrm{s}}$). We compute the variability of style representations among fragments and take its mean across all samples:

$$ $\mathcal{V}_{\mathrm{f}}^{\mathrm{s}}=\frac{1}{N}\sum_{i=1}^{N}\nu_{j=1}^{L}({ \bm{z}}^{\mathrm{s}}_{ij})\text{.}$ (7) $$

Style variability across samples ($\mathcal{V}_{\mathrm{s}}^{\mathrm{s}}$). We compute the average style representation along the fragment axis and measure its variability along the sample axis:

$$ $\mathcal{V}_{\mathrm{s}}^{\mathrm{s}}=\nu_{i=1}^{N}\bigl{(}\frac{1}{L}\sum_{j= 1}^{L}{\bm{z}}^{\mathrm{s}}_{ij}\bigr{)}\text{.}$ (8) $$

### 3.3 Variance-Versus-Invariance (V3) Constraints

With the variability statistics, we can formalize the general relationship between content and style along the sample or fragment axis:

- •
Content should be more variable within samples than across samples, i.e., $\mathcal{V}_{\mathrm{f}}^{\mathrm{c}}\gg\mathcal{V}_{\mathrm{s}}^{\mathrm{c}}$.
- •
Style should be more variable across samples than within samples, i.e., $\mathcal{V}_{\mathrm{s}}^{\mathrm{s}}\gg\mathcal{V}_{\mathrm{s}}^{\mathrm{f}}$.
- •
Within a sample, content should be more variable than style, i.e, $\mathcal{V}_{\mathrm{f}}^{\mathrm{c}}\gg\mathcal{V}_{\mathrm{f}}^{\mathrm{s}}$.
- •
Across samples, style should be more variable than content, i.e., $\mathcal{V}_{\mathrm{s}}^{\mathrm{s}}\gg\mathcal{V}_{\mathrm{s}}^{\mathrm{c}}$.

We quantify the above contrasts as regularization terms, using the hinge function to cut off gradient back-propagation when the ratio between two variability statistics reaches a certain threshold $r>1$, which stands for relativity :

$$ $\displaystyle\mathcal{L}_{\mathrm{content}}$ $\displaystyle=\max(0,1-\frac{\mathcal{V}_{\mathrm{f}}^{\mathrm{c}}}{r\cdot \mathcal{V}_{\mathrm{s}}^{\mathrm{c}}})\text{,}\quad\text{(}\mathcal{V}_{ \mathrm{f}}^{\mathrm{c}}\gg\mathcal{V}_{\mathrm{s}}^{\mathrm{c}}\text{)}$ (9) $\displaystyle\mathcal{L}_{\mathrm{style}}$ $\displaystyle=\max(0,1-\frac{\mathcal{V}_{\mathrm{s}}^{\mathrm{s}}}{r\cdot \mathcal{V}_{\mathrm{f}}^{\mathrm{s}}})\text{,}\quad\text{(}\mathcal{V}_{ \mathrm{s}}^{\mathrm{s}}\gg\mathcal{V}_{\mathrm{f}}^{\mathrm{s}}\text{)}$ (10) $\displaystyle\mathcal{L}_{\mathrm{fragment}}$ $\displaystyle=\max(0,1-\frac{\mathcal{V}_{\mathrm{f}}^{\mathrm{c}}}{r\cdot \mathcal{V}_{\mathrm{f}}^{\mathrm{s}}})\text{,}\quad\text{(}\mathcal{V}_{ \mathrm{f}}^{\mathrm{c}}\gg\mathcal{V}_{\mathrm{f}}^{\mathrm{s}}\text{)}$ (11) $\displaystyle\mathcal{L}_{\mathrm{sample}}$ $\displaystyle=\max(0,1-\frac{\mathcal{V}_{\mathrm{s}}^{\mathrm{s}}}{r\cdot \mathcal{V}_{\mathrm{s}}^{\mathrm{c}}})\text{.}\quad\text{(}\mathcal{V}_{ \mathrm{s}}^{\mathrm{s}}\gg\mathcal{V}_{\mathrm{s}}^{\mathrm{c}}\text{)}$ (12) $$

We obtain the V3 regularization term (used in Equation 1) by summing up the four terms:

$$ $\mathcal{L}_{\mathrm{V3}}=\mathcal{L}_{\mathrm{content}}+\mathcal{L}_{\mathrm{ style}}+\mathcal{L}_{\mathrm{fragment}}+\mathcal{L}_{\mathrm{sample}}\text{.}$ (13) $$

## 4 Experiments

We evaluate V3 on two tasks of different domains to demonstrate its effectiveness and generalizability. The first task is learning pitches and timbres from monophonic music audio (Section 4.2), and the second task is learning digits and ink colors from written digit strings (Section 4.3). The highlight of this section is that V3 effectively learns disentangled representations of content and style, and the discrete content representations align well with human knowledge.

### 4.1 Experiment Setup

Baselines: We compare V3 with two unsupervised baselines: 1) an unsupervised content-style disentanglement based on MINE , and 2) a 2-branch autoencoder similar to our architecture choice, but trained with the cycle consistency loss after decoding and encoding shuffled combinations of $\tilde{{\bm{z}}^{\mathrm{c}}}$ and ${\bm{z}}^{\mathrm{s}}$ .

Additionally, we compare with two supervised learning methods. The first one is a weakly supervised method provided with content labels, in which the model is trained to predict the correct content labels from ${\bm{z}}_{ij}^{\mathrm{c}}$ as a replacement of the VQ layer, and the decoder is trained to reconstruct inputs from ${\bm{z}}_{ij}^{\mathrm{s}}$ and ground truth content labels ; . The second one is a fully supervised method provided with both content and style labels, in which the model learns to predict both content and style from their latent representations.
We provide further details of the baselines in the appendix.

**Table 1: Evaluation of pitch and timbre disentanglement and pitch codebook interpretability. The metrics are defined in Section 4.1.**
| Method | $K$ | Content | Style | Codebook Interpretability |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PR-AUC $\uparrow$ | Best F1 $\uparrow$ | PR-AUC $\uparrow$ | Best F1 $\uparrow$ | Accuracy $\uparrow$ | $\sigma$ $\downarrow$ |  |  |
| V3 | $12$ | 0.904 | 0.904 | 0.879 | 0.879 | 0.999 | 0.001 |
| $24$ | 0.794 | 0.805 | 0.704 | 0.723 | 0.929 | 0.022 |  |
| $48$ | 0.723 | 0.752 | 0.757 | 0.807 | 0.901 | 0.045 |  |
| MINE-based | $12$ | 0.085 | 0.147 | 0.108 | 0.152 | 0.138 | 0.038 |
| $24$ | 0.092 | 0.153 | 0.160 | 0.189 | 0.294 | 0.089 |  |
| $48$ | 0.109 | 0.165 | 0.186 | 0.206 | 0.239 | 0.006 |  |
| Cycle loss | $12$ | 0.083 | 0.146 | 0.113 | 0.164 | 0.275 | 0.104 |
| $24$ | 0.082 | 0.145 | 0.151 | 0.221 | 0.285 | 0.119 |  |
| $48$ | 0.084 | 0.150 | 0.200 | 0.241 | 0.182 | 0.062 |  |
| Weakly supervised | $12$ | 0.730 | 0.759 | 0.704 | 0.725 | 0.999 | 0.001 |
| Fully supervised | $12$ | 0.904 | 0.904 | 0.905 | 0.904 | 1.000 | 0.000 |

Ablation: For ablation, we experiment with another type of variability measurement $\nu_{k}(\cdot)$, which is standard deviation (SD). Besides, we train four ablated versions of V3, each without one of the four regularization terms defined in Equation 9-12 to test its robustness.

Choice of $K$: In reality, we do not always know the real number of content labels, and there is a ubiquity of vocabulary redundancy in human symbol systems, including language. This fact marks an improvement of our method from classification-based methods that we do not need to know the real value of $K$ to perform well. To fully illustrate this, we test all unsupervised methods under three different $K$ settings, which stand for different levels of codebook redundancy.

Evaluation Metrics: We evaluate the models both quantitatively and qualitatively, from two aspects:

- •
The content and style disentanglement ability in latent space. For numerical results, we conduct a retrieval experiment to examine the nearest neighbors of every input ${\bm{z}}^{\mathrm{c}}$ and ${\bm{z}}^{\mathrm{s}}$ using ground truth content and style labels, evaluated by the area under the precision-recall curve (PR-AUC) and the best F1 score.
- •
The codebook interpretability. We plot the confusion matrix of codebooks by aligning each codebook atom to one content label. Quantitatively, we compute the codebook accuracy by counting the correct assignments, and the standard deviation ($\sigma$) that shows the degree of disagreement of codebook assignments among different styles. A good codebook that aligns well with human knowledge should have a clean stair-like confusion matrix, with a high accuracy and a low $\sigma$ value.

Figure: Figure 3: t-SNE visualization of the learned pitch (content) and timbre (style) representations. The first row visualizes pitch representations before quantization. The style labels in the second row show the names of instruments.
Refer to caption: x3.png

Figure: Figure 4: Confusion matrices of learned codebooks for pitches. The horizontal axes show pitch labels from “C” to “B”, and the vertical axes show codebook atoms aligned with pitch labels.
Refer to caption: x4.png

### 4.2 Learning Pitches and Timbres from Music

Dataset:
We synthesize a dataset consisting of monophonic music audio of 12 different instruments playing 12 different pitches in an octave from C4 to B4.
Every pitch is played for one second one by one with a random velocity between 80 and 120.
We synthesize the audio at 16kHz for each instrument and further diversify the notes by adding a random amplitude envelope to each note. The audio files are then normalized and processed to magnitude spectrograms.

Experiment Results:
We present the quantitive results on the test set in Table 1. We see that V3 exhibits superior performance in terms of both representation disentanglement and codebook interpretability compared to the unsupervised baselines, regardless of the codebook size $K$. It is worth noticing that V3 also outperforms the weakly supervised baseline in the retrieval of timbre (style) representation task, which indicates that V3 learns better-disentangled timbre representations containing less pitch information.

Figure 3 further shows the t-SNE visualization of the pitch and timbre representations by different methods when $K=12$, demonstrating that V3 learns representations with clean clustering comparable to supervised methods. Figure 4 shows the confusion matrices of content codebooks of classification results when $K=12$. Among unsupervised methods, V3 achieves the clearest one-to-one mapping from 12 codebook vocabularies to 12 pitches, illustrating the strong symbolic-level interpretability of learned contents.

**Table 2: Ablation of different V3 settings on learning pitches and timbres.**
| Method | $K$ | Content | Style | Codebook Interpretability |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PR-AUC $\uparrow$ | Best F1 $\uparrow$ | PR-AUC $\uparrow$ | Best F1 $\uparrow$ | Accuracy $\uparrow$ | $\sigma$ $\downarrow$ |  |  |
| V3 | $12$ | 0.904 | 0.904 | 0.879 | 0.879 | 0.999 | 0.001 |
| V3 ($\nu_{k}=\mathrm{SD}$) | $12$ | 0.138 | 0.183 | 0.241 | 0.330 | 0.333 | 0.101 |
| V3 (w/o $\mathcal{L}_{\mathrm{content}}$) | $12$ | 0.210 | 0.285 | 0.677 | 0.696 | 0.436 | 0.034 |
| V3 (w/o $\mathcal{L}_{\mathrm{style}}$) | $12$ | 0.899 | 0.903 | 0.810 | 0.817 | 0.999 | 0.001 |
| V3 (w/o $\mathcal{L}_{\mathrm{fragment}}$) | $12$ | 0.276 | 0.360 | 0.543 | 0.562 | 0.455 | 0.074 |
| V3 (w/o $\mathcal{L}_{\mathrm{sample}}$) | $12$ | 0.903 | 0.903 | 0.509 | 0.570 | 0.997 | 0.001 |

**Table 3: Evaluation of digit and color disentanglement and digit code book interpretability. The metrics are defined in Section 4.1.**
| Method | $K$ | Content | Style | Codebook Interpretability |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PR-AUC $\uparrow$ | Best F1 $\uparrow$ | PR-AUC $\uparrow$ | Best F1 $\uparrow$ | Accuracy $\uparrow$ | $\sigma$ $\downarrow$ |  |  |
| V3 | $10$ | 0.734 | 0.793 | 0.967 | 0.937 | 0.892 | 0.006 |
| $20$ | 0.938 | 0.902 | 0.945 | 0.916 | 0.997 | 0.018 |  |
| $40$ | 0.843 | 0.824 | 0.953 | 0.930 | 1.000 | 0.045 |  |
| MINE-based | $10$ | 0.218 | 0.308 | 0.355 | 0.386 | 0.409 | 0.081 |
| $20$ | 0.117 | 0.186 | 0.226 | 0.335 | 0.256 | 0.098 |  |
| $40$ | 0.134 | 0.192 | 0.268 | 0.363 | 0.506 | 0.062 |  |
| Cycle loss | $10$ | 0.143 | 0.184 | 0.244 | 0.288 | 0.710 | 0.037 |
| $20$ | 0.560 | 0.634 | 0.237 | 0.278 | 0.896 | 0.043 |  |
| $40$ | 0.507 | 0.587 | 0.219 | 0.257 | 1.000 | 0.041 |  |
| Weakly supervised | $10$ | 0.952 | 0.951 | 0.575 | 0.570 | 1.000 | 0.000 |
| Fully supervised | $10$ | 0.952 | 0.951 | 0.962 | 0.915 | 1.000 | 0.000 |

**Table 4: Ablation of V3 settings on learning digits and colors.**
| Method | $K$ | Content | Style | Codebook Interpretability |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PR-AUC $\uparrow$ | Best F1 $\uparrow$ | PR-AUC $\uparrow$ | Best F1 $\uparrow$ | Accuracy $\uparrow$ | $\sigma$ $\downarrow$ |  |  |
| V3 | $10$ | 0.734 | 0.793 | 0.967 | 0.937 | 0.892 | 0.006 |
| V3 ($\nu_{k}=\mathrm{SD}$) | $10$ | 0.427 | 0.532 | 0.497 | 0.518 | 0.648 | 0.051 |
| V3 (w/o $\mathcal{L}_{\text{content}}$) | $10$ | 0.453 | 0.525 | 0.926 | 0.890 | 0.586 | 0.051 |
| V3 (w/o $\mathcal{L}_{\text{style}}$) | $10$ | 0.634 | 0.675 | 0.918 | 0.888 | 0.726 | 0.035 |
| V3 (w/o $\mathcal{L}_{\text{fragment}}$) | $10$ | 0.977 | 0.993 | 0.953 | 0.909 | 1.000 | 0.000 |
| V3 (w/o $\mathcal{L}_{\text{sample}}$) | $10$ | 0.480 | 0.578 | 0.942 | 0.894 | 0.653 | 0.024 |

Ablation Study:
We compare the performances of V3 and its variants in Table 2. It can be seen that $\nu_{k}=\mathrm{SD}$ does not work as well as $\nu_{k}=\mathrm{MPD}$ due to its weakness in constraining multi-peak content distributions within samples. It is also worth noting that V3 performs fairly well even when discarding $\mathcal{L}_{\mathrm{style}}$ or $\mathcal{L}_{\mathrm{sample}}$. In both cases, we observe a decrease in the discarded loss even if we do not explicitly optimize for it. We suspect this is due to the robustness of V3 constraints as reflected in the symmetric relationships among the four losses—we can enforce three relations, and the fourth one may fall into the right place. However, in practice, it is difficult to tell the “free relation” beforehand as it is also related to detailed content and style variations in specific domains.

### 4.3 Learning Digits and Colors from Written Digit Strings

Dataset:
We synthesize an image dataset of written digit strings on light backgrounds using all 10 digits and 8 different ink colors. The order of digits is random. All images are diversified with Gaussian noises, random blur, and foreground and background color jitters.
More details about the dataset can be found in the appendix.

Figure: Figure 5: t-SNE visualization of the learned digit (content) and color (style) representations. The first row shows content representations before quantization, and the second row’s visualizations use the same colors as the digits are written.
Refer to caption: x5.png

Figure: Figure 6: Confusion matrices of learned codebooks for digits. The horizontal axes show digit labels from “0” to “9”, and the vertical axes show codebook atoms aligned with digit labels.
Refer to caption: x6.png

Experiment Results:
The quantitive results can be seen in Table 3. Similar to the music task, V3 performs the best considering both representation disentanglement and codebook interpretability. Even though some unsupervised baselines also achieve a high codebook accuracy at large $K$ values, they fall behind in terms of latent space retrieval, especially in style. We visualize the latent representations in Figure 5, and compare the confusion matrices in Figure 6 for intuitive understanding.

Ablation Study: Table 4 shows the ablation results of V3 on the digit string dataset. The inferior performance of $\nu_{k}=\mathrm{SD}$ supports our assumption of its weakness in capturing multi-peak high-dimensional distributions. We observe that V3 trained without $\mathcal{L}_{\mathrm{fragment}}$ achieves the best codebook interpretability. Referring to the ablation in section 4.2, it is conceivable that a three-loss target can sometimes also satisfy the four constraints as defined in V3, and the optimal value for hyperparameters r and $\beta$ may differ by datasets, which marks a possible direction for future improvements.

## 5 Limitation

We have identified several limitations in our V3 method that necessitate further investigation. First, while V3 achieves good disentanglement and symbolic interpretability, it is not flawless — samples of different contents (say images of “8" and “9") may sometimes be projected into the same latent code. Inspired by human learning, which effectively integrates both mode-1 and mode-2 cognitive processes, we aim to enhance V3 by incorporating certain feedback or reinforcement. This adaptation could also facilitate the application of V3 to more complex domains such as general image or video. Additionally, V3 is currently optimized to disentangle content and style from data samples that include defined fragments. Extending this capability to unsegmented data, such as continuous audio, represents a significant area for future development. Furthermore, V3 assumes that content elements do not overlap, which does not hold in cases of polyphonic music or mixed audio. Addressing this challenge will require a more sophisticated approach that considers the hierarchical nature of content.

## 6 Conclusion

In conclusion, we contributed an unsupervised content-style disentanglement method V3, which leads to an emergence of symbolic-level interpretability of the learned latent space. V3’s inductive bias is domain-general, intuitive, and concise, solely based on the meta-level insight of the statistical difference between content and style, i.e., their distinct variance-invariance patterns reflected both within and across data samples. Experiments show that V3 not only outperforms the baselines in terms of both disentanglement and latent space interpretability but also achieves comparable results with supervised models under a similar auto-encoding architecture. Moreover, the effectiveness of V3 generalizes across two distinct domains: music audio and written digits. For contents, the VQ codebooks learned by V3 have a near one-to-one alignment with human knowledge of music scales (the 12 semitones from C to B) and numbers (the 10 digits from 0 to 9). For style, V3 learns a latent space where different colors or instruments naturally form clusters without any label supervision or even a KL regularization that is usually needed for a well-structured latent space.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: - How does the choice of codebook size (K) affect interpretability and disentanglement performance?  

- The method uses specific values for the relativity parameter r and the V3 loss weight beta  depending on the dataset. Could you clarify the intuition behind selecting these values? How sensitive the performance is to different values of these hyperparameters?

- The paper mainly utilizes small datasets with specific content-style boundaries. How does it scale with larger datasets?

Minor issues: 
Line 463: The term "contennt" should be corrected to "content."
soundness: 3
strengths: ++ Unsupervised approach for separating style and content, which is a plus for domains where labeled data is unavailable or scarce

++ Generalizes across audio, image and video data. 

++ Content representations show interpretable properties. 

++ Performs well for out of distribution styles
summary: They propose an unsupervised method to disentangle content and style representations.  They use vector-quantized autoencoder architecture and incorporate variance-versus-invariance constraints. Their method benefits from meta-level inductive bias, and shows out-of-distribution generalization. They demonstrate their method’s effectiveness across diverse applications such audio, images and video clips. Moreover, the learned codebook enables interpretable features.
weaknesses: -- They assume content elements are distinct, which can limit its application in domains with overlapping content.

-- The method is optimized for data with predefined fragments, which may not generalize well to continuous or unsegmented data types.

-- The experiments are done on relatively small datasets.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. It is better to evaluate V3 on  more complex or challenging datasets, such as face datasets FFHQ? And It is better to add discussion on 
 potential limitations of V3 when dealing with more complex data containing multiple contents or styles within a single sample.
2. Since V3 uses different losses to control content and style representations through self-supervised means, is there a risk that the representations intended to learn style might inadvertently capture content instead? How can it be ensured that there is not too much coupling between content and style representations? I suggest to conduct an analysis showing how much content information can be recovered from the style representations and vice versa.
3. Why is it necessary to apply VQ to content representations to form a codebook? What would be the impact on the learning results if VQ were not used? Seeing corresponding results in the Ablation study could be more convincing. 
4. Are the numbers of content and style representations in V3 the same? Considering that the actual samples may have varying quantities suitable for representing content and style, would setting different numbers for content and style representations in the same training round make a significant difference in the results?
soundness: 2
strengths: 1. The V3 method offers a novel unsupervised disentanglement approach that leverages the universal statistical differences between content and style. This is a creative application of variance-invariance constraints within an encoder-decoder framework, distinct from traditional methods that rely on supervision or paired data.
2. The loss calculation method proposed by V3, based on content and style, is concise and clear, aligning with human cognitive characteristics. Humans also tend to keep the style of the same sample similar while focusing on the content differences across different samples.
3. The model proposed in the paper is applicable to the learning of multiple modalities of style and content, including images, music, and animations, demonstrating strong generalizability. Moreover, the loss calculation pattern presented is not limited to the paper model but can also be applied to other related models for style and content learning.
4. The paper is well-structured, with a clear problem statement, methodology, and experimental evaluation. The use of datasets from multiple domains enhances the robustness of the research findings. The logic is strong, and the hierarchy is clear. The visualization of learned representations also helps readers better understand the results.
summary: This paper introduces an unsupervised method named V3 (variance-versus-invariance) for disentangling content and style representations from a sequence of observations. The method is based on the assumption that content varies significantly within a sample but maintains a consistent vocabulary across samples, while style remains relatively invariant within a sample but varies significantly across samples. V3 integrates this inductive bias into an encoder-decoder architecture and demonstrates its effectiveness across multiple domains and modalities, including music audio, handwritten digit images, and simple animations. The paper reports that V3 outperforms existing unsupervised methods and even surpasses supervised methods in out-of-distribution generalization and few-shot learning. Additionally, V3 exhibits symbolic-level interpretability, aligning machine representations closely with human knowledge.
weaknesses: 1. The paper's approach to learning sample content and style is reliant on datasets with a specific structure, which may limit its effectiveness when applied to images with varied or complex styles. For example, the Written Phone Numbers dataset contains only one digit in a specific position on each image, but the style of the digit can vary; in SPRITES, the character's position is fixed, and the form is also relatively uniform. If an image contains more content or more complex content, the model may have difficulties learning content and style.
2. The Ablation study reveals that certain components of the loss function may adversely affect performance metrics, suggesting a need for a deeper investigation into the roles and interactions of these components within V3.
3. It appears that V3 may have difficulty distinguishing between some similar contents during recognition. Similarly, although the paper may not mention it, V3 may also have similar problems with distinguishing styles, thus limiting the style discrimination between different samples.
4.Some important related works[a,b] are missing, which are also rely one some assumption to disentangle content and style.   The detailed comparison should be included, especially on the basic assumptions and the scope of suitable cases. 
a. Retriever: Learning Content-Style Representation as a Token-Level Bipartite Graph
b. Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: See above.
soundness: 3
strengths: 1. The proposed method does not rely on external supervision such as labels.
2. The proposed method can generalize to unseen data.
3. The disentangled content representations align with human perceptions in some cases.
summary: The paper proposes an unsupervised method that learns from raw observation and disentangles its latent space into content and style representations based on the meta-level insight of domain-general statistical differences between content and style.
weaknesses: 1. The paper does not explicitly discuss how the codebook size and codebook dimension are determined. This could limit the application of the proposed method to different datasets.
2. The experiments are all conducted on toy datasets. This weakens the method's practical use in real-world scenarios. For example, for audio disentanglement, the musical note disentanglement is too simple. It is recommended to perform real-world disentanglement experiments such as voice conversion using VCTK.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Can the author elaborate a bit more on the experiment setting of Section 4.2?
'we evaluate the models’ content-style disentanglement ability by conducting a retrieval experiment to examine the nearest neighbors of every input zc and zs using ground truth content and style labels, evaluated by the area under the precision-recall curve (PR-AUC) and the best F1 score'
soundness: 3
strengths: 1. This paper proposes an innovative disentanglement algorithm that does not rely on external labels of the style or content, making a novel contribution over the existing work, most of which requires external labels.

2. There is a clear rationale behind the proposed algorithm, and the rationale is interesting

3. The presentation in the paper is very clear.
summary: This paper proposes an unsupervised disentanglement algorithm between styles and content, based on the assumption that style information remains consistent within samples, but varies across samples; whereas content information varies within samples, but the aggregated content is stable across samples due to the fixed vocabulary.
weaknesses: 1. Although the proposed method does not rely on any external labels, it seems to be built upon rather strong assumptions. In particular, the method assumes that the aggregated content of a sample should stay constant across different samples, at least more stable than the style. This assumption is very strong, and there are many scenarios where this may not hold. First, if the content information in each example is too small to exhaust all the possible combinations, then the content would still vary greatly across samples. For example, in the written digit case, if each example only contains one digit. Then the model may no longer be very successful. This limitation may also hold for many other scenarios, such as Chinese calligraphy, which typically only contains very few characters, paintings containing only one object, etc. Second, if the content information exhibits too much variability, then the content across different statements may still have more variability than style. For example, consider short passages of different languages, where the content is the meaning of the passage and the style is language. The content in this case still varies greatly across different passages. Meanwhile, the authors also acknowledge that the method cannot deal with large content vocabularies. This limitation, together with the limitation discussed above, significantly limits the applicability of the proposed method.

2. Related to the first limitation, the experiment section seems to focus on very simple, carefully crafted toy scenarios that happen to satisfy the assumptions of the proposed method. There is no experiment on real-world scenarios, such as disentangling real human speech, photos, paintings, music, passages, etc. The lack of real-world experiments adds to the concern that the assumptions made by the proposed method may deviate too much from real-world cases to render the method useful.

3. (Minor) On page 5, the authors state that 'Theoretically, we aim to measure the consistency of codebook usage distribution along the sample axis, which is not differentiable'. This is not necessarily true. We can derive a soft, differentiable distribution of codebook usage via a softmax codebook selection. That is, for each codebook selection, rather than have a one-hot vector indicating which code is selected, we can derive a softmax selection, which is very common in deep-learning-based vector quantization methods, such as VQ-VAE. As a result, I believe that using the mean content vector to represent the distribution information may be improved.

4. (Minor) In line 242, the authors state that 'Content should be more variable within samples than across samples'. This is an overly simple statement. A more precise version should be 'Content should be more variable within samples than the sample-wise aggregated content across samples'.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: none
soundness: 3
strengths: The paper is well written and easy to follow. The method is well motivated and described and results are nicely presented.
summary: The authors propose a method to learn representations from an adapted VQ-VAE that disentangle content and style based on their different statistical characteristics: style is more constant over patches of a sample, whereas content can vary.
weaknesses: I have few comments on the main body of the paper, so please do not see this as a "few line review", I just do not have much to say.

**Experiments**: 
* 3 of 4 are on synthetic data with only SVHN being a "natural" data set and even that is cropped to the task. 
* It is of course right to demonstrate that the model performs well on data that follows its assumptions. 
* However, it would be informative (a) to see results on more natural datasets, and (b) to see how the model performs where assumptions do not hold so well, if only as a benchmark for future improvement.

Meta-review: This paper introduces V3 (variance-versus-invariance), an unsupervised method designed to effectively disentangle content and style representations from sequences of observations without relying on domain-specific labels or knowledge.

The reviewers acknowledged the novelty and contribution of this work for the V3 method's ability to separate style and content without relying on external labels or paired data, making it highly advantageous for domains with scarce or unavailable labeled data (Reviewer GuWP and ipXQ). In addition, reviewers highlighted V3's robust performance across diverse data types, including audio, images, and video (Reviewer GuWP and D5wf).

Some reviewers also raised concerns that V3 assumes content elements are distinct and consistent across samples (Reviewer GuWP and ipXQ). Moreover, it was noted that V3 is optimized for data with predefined fragments, potentially limiting its performance on continuous or unsegmented data types (Reviewer GuWP and D5wf). Last but not least, almost all the reviewers mentioned the reliance on relatively small or toy datasets, such as handwritten digits and simple animations, which raises concerns about the method's practical applicability in real-world scenarios.

During the rebuttal, the authors made great efforts to address the reviewers’ concerns. The main concern about the assumption and lacking experiments on a larger dataset, had been well addressed by the additional clarifications provided by the authors. Some other minor concerns are acknowledged by the authors and claimed to be regarded in future work. Considering all factors, the AC therefore recommends acceptance of this paper.

Decision: Accept (Poster)