Learning Interpretable Low-dimensional Representation via Physical Symmetry
Authors: Xuanjie Liu
Daniel Chin
Yichen Huang
Gus Xia
Venue: NeurIPS 2023 poster (2023)
Keywords: Physics Symmetry
Time series data
Self-supervised Learning
Representation Augmentation
TL;DR: This study uses physics symmetry as an effective inductive bias to learn interpretable representations from time-series data in a self-supervised fashion.

Abstract: We have recently seen great progress in learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord and texture. However, most methods rely heavily on music domain knowledge. It remains an open question what general computational principles *give rise to* interpretable representations, especially low-dim factors that agree with human perception. In this study, we take inspiration from modern physics and use *physical symmetry* as a self-consistency constraint for the latent space. Specifically, it requires the prior model that characterises the dynamics of the latent states to be *equivariant* with respect to certain group transformations. We show that physical symmetry leads the model to learn a *linear* pitch factor from unlabelled monophonic music audio in a self-supervised fashion. In addition, the same methodology can be applied to computer vision, learning a 3D Cartesian space from videos of a simple moving object without labels. Furthermore, physical symmetry naturally leads to *counterfactual representation augmentation*, a new technique which improves sample efficiency.

Learning Interpretable Low-dimensional Representation via Physical Symmetry
Xuanjie Liu
Daniel Chin 1.2
Yichen Huang
Gus Xia 1.2
MMohamed bin Zayed University of Artificial Intelligence
Neww York University Shanghai
(Xuanjie.Luu, Nanfeng. Qin, Yichen. Huang, Gus.Xia) mbzzuai.ac.a
Abstract
We have recently seen great progress learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord and texture. However, most methods rely heavily on music domain knowledge. It remains open question what general computational principles give rise to interpretable representations, especially low-dim factors that agree with human perception. In this study, we take inspiration from modern physics and use physical symmetry as a self-consistency constraint for the latent space of time-series data. Specifically, it requires the prior model that characterises the dynamics of the latent states to be equivariant with respect to certain group transformations. We show that physical symmetry leads the model to learn linear pitch factor from unlabelled monophonic music audio in a self-supervised fashion. addition, the be applied computer vision, learning 3D) Cartesian space from videos of a simple moving object without labels. Furthermore, physical symmetry naturally leads counterfactual representation augmentation, new technique which improves sample efficiency.
methodology
Introduction
Interpretable representation--larniii models have achieved great progress for various types of time- series data. Taking the music domain as an example, tailored models [Ji el al. 2020 have been developed learn pitch, timbre, melody contour, chord progression, texture, etc.
music audio.
These human-interprette
representations have greatly improved the performance of generative algorithms in various music creation tasks, including inpainting (Wei al. [2022. harmonization YY)
el al. 2022). (re-) rre))arrangemen,, and performance rendering [Jeong et al. 2019].
However, most representation learning models still rely heavily domain-specee knowledge. For example, to use pitch scales or instrument labels for learning pitch and timbre representations (Luo]
al. 2020 2019 [nngel al.) 2020 Lin al. 2021 [sling et al. [2018] and use chords and rhythm labels for learning higher-level representations AAama| 2019 Yang et al.) 2019: Wang el
al. 2020, Wei and Xia 2021] Such approach is presumably very different from human learning:
even without formal music training, we see that many people can learn pitch, a fundamental music concept. simply from the experience listening music. Hence, remains an open question how to learn interpretable pitch factor using inductive biases that are more general. In other words, what general computational principle gives rise to the concept of pitch.
We see a similar issue in other domains. For instance, various computer-vision models (McCarthy and Ahmed, 2020 Trevithick and Yang 2021 Mescheder et al. 2019 Riegler et al. 2017 can learn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

3D representations of human faces or a particular scene by using domain knowledge (e.g.. labelling of meshes and voxels, 3D convolution, etc.) But when these domain setups are absent, it remains non-trivial task to learn the 3D location of simple moving object in self-supervised fashion.
E
D
7tt1.
S
*1+1
E
S S-1
Zt
D
Figure I: An illustration of physical symmetry our inductive bias.
Inspired by modemn physics, explore to use physical symmetry (i.e., symmetry of physical laws) as a weak selficooni constraint for the learned latent & space of time-series data æ. As indicated in Figure [1 this general inductive bias requires the learned prior model R, which is the induced physical law describing the temporal flow of the latent states, to be equivariant to a certain transformation $ (e.g., translation rotation). Formally, = R(z,) if and only if 2 R(2f), ). where N S(2). In other words, R and S are commutable for 2, i.e., R(S(z))= = S ((()))) Note that our equivariant
assumption applies only the latent space. This is fundamentally different from most existing
symmetry-informel models [Bronstein et al. 2021 in which the equivariant property also imposes assumptions on the raw data space.
Specifically, we design self-supervised learning with physical symmetry (SPS a method that adopts encoder-decoder framework and applies physical symmetry to the prior model. We show that SPS learns a linear pitch factor (that agree with human music perception) from monophonic music audio without any domain-specifff knowledge about pitch scales, f0, or harmonic series. The same methodology can be applied to the computer vision domain, learning 3D Cartesian space from monocular videos of bouncing ball shot from fixed perspective. In particular, properties SPS as a self-supervised algorithm for interpretability:
four desired
Conciseness: SPS does not require contrastive samples, bach normalization, or a large batch size. We can even drop the Gaussian prior regularization, which is usually required to learn a meaningful latent space. (See Section
3.2
Section
Sample efficiency: SPS learns low low-dimensional representations of a dynamic system in very sample-efficiee way. (See Section Section 5.2)
Robustness: Even with an incorrect symmetry assumption, SPS can still learn more inter- pretable representations than baselines. (See Section (5.3)
Extendability: SPS can be easily combined with other learning techniques. For example, we further assume extra global invariant style code, the model becomes disentangled sequential autoencoder, capable of learning content-style disentanglement from temporal signals. (See appendix.)
2 Intuition
The idea of using physical symmetry for representation lcarning comes from modern physics. In classical physics, scientists usually first induce physical laws from observations and then discover symmetry properties the law. (E.g.. Newton's 's law of gravitation, which was induced from planetary orbits, is symmetric with respect to Galilean transformations.) In contrast, in modern physics, scientists often start from a symmetry assumption, based on which they derive the corresponding law and predict the properties (representations) of fundamental particles. (E.g., general relativity was developed based on a firm assumption of symmetry with respect to Lorentz transformations).
'The source code is publicly available at bttp://giiibbbcom///uuuiiiiliiip www.sssppriiidddddiinneeeee via The demo page available at htttps///uumjiedinggiibbii... demo/

---

Analogously, we use physical symmetry as an inductive bias of our representation learning model, which helps us learn a regularised prior and an interpretable low-dim latent space. If belief of many physicists that symmetry in physical law is a major design principle of nature, we regard symmetry in physical law as general inductive bias of perception. In other words, physical symmetry leads an Al agent to learn human-aligned concepts in self-supervised fashion, we believe that it could also provide insights into the ways that human minds perceive the world.
The introduction of physical symmetry naturally leads to counterfactual representation augmenta- tion, a novel learning technique which helps improve sample efficiency. Representation augmentation means to "imagine" extra pairs of zi and 2 as training samples for the prior model R. Through the lens of causality, this augmentation can be seen as a counterfactual inductive bias of the prediction on the representation level - what if the prior model makes predictions based on transformed latent codes? As indicated Figure [ it requires the prediction of the z sequence to equivariant to certain group transformations, S. This regularisation also constrains the encoder and decoder indirectly through the prior model since the network is trained an end-to-end fashion.
Methodology
With physical symmetry, we aim learn an interpretable low-dimensional representation 2, of each high-dimensione sample z; from time-series X1:7. We focus on two problems in this paper: 1) to learn /D linear pitch factor of music notes from music audio, where each x; spectrogram of note, and 2) to learn 3D Cartesian location factors of a simple moving object (a bouncing ball) from its trajectory shot by fixed, single camera, where each 2; an image.
3.1 Model
Figure 2 shows the model design of SPS. During the training process, the temporal data input X1.7 is first fed into the encoder E to obtain the corresponding representation Z1:P. Then it is fed into three branches. In the first branch (the green line), 41:T is decoded directly by the decoder D to reconstruct xy,T- In the second branch (the orange line), Zır is passed through the prior model R to predict its next timestep. 22. which is then decoded to reconstruct N2:7+1- In the third branch (the blue line), transform Z1:7 with S, pass it through R, and transform it back using the inverse transformation S-1 to predict another version of the next timestep 22.4411 and finally decode it to 82.7+1. We get three outputs from the model: X1.r. X and X2:T+1-
The underlying idea of physical symmetry that the dynamics of latent factor and its transformed version follow the same physical law characterised by R. Therefore, ž and 2 should be close each other and so are x and &, assuming S a proper transformation. This self-consistency constraint helps the network learn a more regularised latent space.
21:77
R
22.741
D
82r+1
Z2.T+1
82:7+1
X1.r
Zi:r
R
2
Figure 2: An overview of our model. XI.T is fed into the encoder E to obtain the corresponding representation Zi, which then fed into three different branches yielding three outputs respectively: Xi.r, S2.41 and N2.4+1- Here, R. is the prior model and S is the symmetric operation. The inductive bias of physical symmetry enforees R to be equivaraint with respect to S, so and 2 should close to each other and so are and &.
3

---

3.2 Training objective
The total loss contains four terms: reconstruction loss Leec prior prediction loss Lpriors symmetry- based loss Lsym, and KI. divergence loss LKLD. Formally,
= fee + X1 Eprior + Xy + >3
where A1, 22 and A3 are weighting parameters. By referring to the notations in section 3.1
Lee LECE(X XIT) + LBCE (X2.7 X2:r) + LBCE (XCE(X2:7... (2:T,Xx2771
(2)
Ppaue (2(22 (2:r),
(3)
Syym lo (22:7 2o:T) + l2 (Z2.7 Z2.r).
(4)
CKLD the Kulback-Leibee divergence loss between the prior distribution of 2; and standard Gaussian. Lastly, we build two versions of SPS: SPSyAE and SPS with the latter replacing the VAE with an AE (and trivially doesn't have LKLD)).
3.3 Symmetry-based counterfactual representation augmentation
During training, S is the counterfactual representation augmentation since it creates imaginary sequences of = (i.e., imaginary experience) to help train the prior. In practice, for each batch we apply K different transformations S and yield K imaginary sequences. Thus, the two terms of symmetry-based loss can be specified as:
tg (Z2:T Z2.T) Σ (R(S(((((())) 22:))
(5)
82 (22.T 22.T)
Σ (R(()((()))
(6)
where the lower case denotes the index of specific transformation and we refer to K as the augmentation factor. Likewise, the last term of reconstruction loss can be specified as:
1 K k=1
LBCE(X2:T, (x:))
=
Lucr (D
Each S applied to each sequence Z1r belongs to certain group, and different groups are used for different problems. For the music problem, assume z; be to 1DD and use random S E G = (R, In other words, we add random scalar to the latent codes. As for the video problem, we assume z; be 3D and random S G = (R²2, +) SO(2). In other words, random rotation and translation are applied on two dimensions of z;
Results
We test SPS under two modalities of temporal signals: music (section 4.1 and video (section 4.2) Each model is executed with 10 random initialisations, and evaluated on the test The highlight of this section is that SPS effectively learns interpretable low-dimensional factors that align with human perceptions. Also, by utilizing small training sets, we show the high sampling efficiency of our model. In the appendix, we further show that SPS also maintains accuracy in both reconstruction and prediction tasks (section A.2). Additionally, we present supplementary results trained on more complicated datasets and more advanced configuration of our model, called SPS+, which enables content-style disentanglement in addition to interpretable learning. The findings from these more complex scenarios align closely with those observed in the simpler presented in this section.

---

4.1 Learning linear pitch factor from music audio
4.1.1 Dataset and training setups
We synthesise a training dataset 27 audio clips, each containing 15 notes in major scales with the first 8 notes ascending and the last notes descending. We vary the starting pitch by integer numbers of MIDI pitch such that every MIDI pitch in the range A#4 to C7 is present in the training set. Only the accordion is used to generate the clips. For each clip in the training set, we uniformly randomly shift its pitch upwards by decimal between 0 and in this way generate the test set for evaluation.
We convert each audio clip into a sequence of image segments for processing. First, we run STFT (with sample rate 16000/s, s, window length 1024, hop length 512, with padding and no logarithmic frequency scale) over each audio clip to obtain power spectrogram. After normalising the energy the range [0, 1]. we slice the power spectrogram into fifteen image segments, each containing one note. The CNN encoder, in each timestep, takes one segment as input. For the latent space, we assume z; E R and sample counterfactual representation augmentation ~~ ([[[1,,1] where S E G = (R. +). Note this assumption does not indicate any domain-specinn inductive bias of music, such as the logarithmic relationship between pitch and frequency or the relationship between FO and harmonics.
4.1.2 Results on interpretable pitch space
SPSws (Ouns)
SPS
SPSvaB (Ablation)
SPSAE (Ablation)
&-VAE
SPICE
Embedding
0.8
2
1-
0.6
C3
C6
C3
O6
C5
C6
C3
O6
C6
C3
C8
Pitch
C6
C6
Senthesie
Detectel
0.0
2.5
0.0
2.5
Figure 3: A visualisation of the mapping between the ID learned factor z and the true pitch, in which straight lines indicates better result. In the upper row, models encode notes in the test to The x axis shows the true pitch and the y axis shows the learned pitch factor. In the lower row, the x axis traverses the z space. The models decode audio clips. We apply YIN to the audio clips to detect the pitch, which is shown by the axis. In both rows, a linear, noiseless mapping is ideal, and our method performs the best.
Figure 3 demonstrates that the ID pitch factor learned by our model exhibits linear relationship with the conventional numerical ordering used to represent pitch by humans (e.g. MIDI pitch numbers). The plot shows the mappings of two tasks and six models. In the embedding task (the first row), the x-axis is the true pitch and the y-axis is embedded z. In the synthesis task (the second row), the r-axis is z and the y-axis is the detected pitch (by YIN algorithm, standard pitch-estimation method by [De Cheveigné and Kawahara 2002 of decoded (synthesised) notes. The first two models are SPS based on VAE and AE, respectively, trained with counterfactual representation augmentation factor K = 4. The third and fourth models are trained without constraints of physical symmetry (K 0), serving as our ablations. The fifth one a vanilla 3 VAE, trained only reconstruct, not to predict. The last one is SPICE [Gfeller et al.) 2020]. SOTA unsupervised pitch estimator with strong domain knowledge on how pitch linearity is reflected in log-frequency spectrugrams. As the figure shows, 1) without explicit knowledge of pitch, our model learns more interpretable pitch factor than 3- 3-V, and the result comparable to SPICE, and 2) without the Gaussian prior assumption of latent variable distribution, our model SPSAE also learns continuous representation space.
5

---

Table I: The linearity of learned pitch factor and synthesized sound pitch evaluated by R..
Methods
Learned factor R² t
Synthesis R²
SPSvAE- K (Ours)
0.99910.001
0.986=10.025
SPSAE:
K 4 (Ours)
0.998++0.001
0.986-00.00
SPSvAE,
K = (Ablation)
0.997++0.002
0.910==0.040
SPSAE:
K=O (Ablation)
0.993+000006
0.832++0.119
8- V-V
0.772=10.333
0.534+10275
SPICE
1.000
N/A
Table 1 shows a more quantitative analysis using R²2 as the metric to evaluate the linearity of the pitch against mapping from the encoder and the decoder. All models except SPICE are trained with 10 random initializations.
4.2 Learning object 3D coordinates from videos of a moving object
4.2.1 Dataset and training setups
We run physical simulations of bouncing ball in 3D space. The ball is randomly thrown and affected by gravity and the bouncing force (elastic force). A fixed camera records a 20-frame video of each 4-second simulation to obtain one trajectory (see Figure 4 The ball's size, gravity, and proportion of energy loss per bounce are constant across all trajectories. For each trajectory, the ball's initial location and initial velocity are randomly sampled. We utilize 512 trajectories for training, and an additional 512 trajectories for evaluation.
For the latent space, we set the dimension of the latent space to 3, but only constrain 2 of them by augmenting the representations with S E = (R.2, +) x SO( Those two dimensions are intended to span the horizontal plane. The third one, which is the unaugmented latent dimension, is intended encode the vertical height.
Figure 4: Two example trajectories from the bouncing ball dataset.
4.2.2 Results on interpretable 3D representation
Figure 5 visually evaluates the interpretability of the learned location factors by traversing the space, one dimension time, and using the learned decoder to synthesise images. the learned factors are linear w.r.t. the 3D Cartesian coordinates, the synthesised ball should display linear motions as we change linearly. In brief, SPS learns an more interpretable and linear space. Here, subplot (a) depicts the results of SPS with K=4. We see that the un-augmented dimension, #2, controls height of the ball, while 21 and 23 move the ball along horizontal (ground) plane. Each axis much more linear than in (b) and (c). Subplot (b) evaluates SPS with counterfactual representation augmentation K=0, essentially turning off SPS. As 2 varies, the ball seems to travel along curves in the 3D space, showing the ablation learns some continuity w.r.t. the 3D space, but is obviously far from linear. In (c), the 3- VAV fails to give consistent meaning to any axis.
Table further shows guantitative evaluations on the linearity of the learned location factor, in which we see that SPS outperforms other models by a large margin. To measure linearity, we linear regression from 2 to the true 3D location over the test set and then compute the Mean Square Errors (MSE). Therefore, a smaller MSE indicates a better fit. To give an intuitive example, the MSEs (a), (b) and (c) in Figure are 0.09. 0.58 and 0.62 respectively. Here, we also include the results of SPS Very similar to the music experiment session 4.1 we again see that even without the Gaussian prior assumption, our model SPS learns an interpretable latent space comparable to SPS

---

(a) SPSvAE: 4 (Ours)
(b) SPSvAB: K-0 (Ahlation)
(c) V-VAE (Baseline)
1.2 -27
125 -20
2
Figure 5: A visualisation of latent-space traversal performed three models: (a) ours, (b) ablation, and (c) baseline, which we see (a) achieves better linearity and interpretability. Here, row shows the generated images when changing 2 and keeping 2 O, where the axis varies 2 from -20, to +2 We center and normalise 2, so that the latent space from different runs is aligned for fair comparison. Specifically, in (a), changing 22 controls the ball's height, and changing 21, 23 moves the ball parallel to the ground plane. In contrast, the behavior in (b) and (c) are less interpretable.
Table 2: Linear fits between the true location and the learned location factor. We run the encoder on the test set to obtain data pairs in the form of (location factor, true coordinates). We then run linear fit on the data pairs to evaluate factor interpretabilityy Two outliers are removed from the 50 runs.
Method
axis MSE
y axis MSE +
axis MSE +
MSE +
SPSvAE: K=4 (Ours)
0.11 + 0.09
0.31 + 0.34
0.26 + 0.34
0.26 + 0.30
SPSAE. K= (Ours)
0.13 + 0.07
0.39 = 0.33
0.21 + 0.17
0.24 + 0.17
SPSvAE. K=0 (Ablation)
0.33 0.10
0.80 = 0.18
0.75 0.17
0.62 0.14
SPSAE. K=0 (Ablation)
0.26 + 0.09
0.44 + 0.27
0.55 0.17
0.42 0.15
3-VAE
0.36 0.03
0.70 + 0.01
0.68 + 0.03
0.58 + 0.01
Analysis
To better understand the effects of counterfactual representation augmentation (first introduced in section (3.3). we ran extra experiments with different S and K. We choose the vision problem since a latent space manifests a more obvious difference when physical symmetry applied. In section 5. we show that a larger augmentation factor K leads to higher sample efficiency. In change of learned latent space against training epoch according to different values of K. In section 5.3 show that some deliberately incorrect group assumptions S can also achieve good results.
section 5.2 we visualise
5.1
Counterfactual representation augmentation improves sample efficiency
Gsows that larger factor of counterfactual representation augmentation leads a lower linear projection loss (the measurement defined in section 4.2.2 of the learned 3D representation. Here, K the augmentation factor, and K means the model is trained without physical symmetry. The comparative study is conducted on 4 training set sizes (256, 512, 1024, and 2048), in which each box plot shows the results of 10 experiments trained with a fixed K and random initialisation. We see that a larger K leads to better results and compensates for the lack of training data. E.g., the loss trained on 256 samples with K = comparable to the loss trained on 1024 samples with and the loss trained 512 samples with K 4 is even lower than the loss trained 2048 samples with K = 0. Furthermore, when K = O, increasing the number training samples beyond a certain point does not further shrink the error, but increasing K still helps.
5.2 Counterfactual representation augmentation improves interpretability
Figure visualises the latent space during different stages of model training, and we see that larger K leads to a better enforcement of interpretability. The horizontal axis shows the training epoch. Three experiments with different K values (x0, x x 16) are stacked vertically. Each experiment is

---

Samples
512 Samples
1024 Samples
2048 Samples
Linear loes 4.
0.6
0.4
0.2
0
0.0
16
16
0
K
Figure 6: comparison of linear projection MSEs among different augmentation factors (K) and training set sizes, which shows that counterfactual representation augmentation improves sample efficiency.
SPSyAE: K=0
SPSyAE: K K-4
SPS VAE, 16
2
-2
-2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
Epoch
331
669
1000
1331
1089
2000
2331
2089
3000
3331
3069
4000
4331
4080
5000
Figure 7: visualisation of the learned latent space against training epoch, in which larger leads to a stronger enforcement on learning a linear latent space. Here, we plot how the encoder projects equidistant 3D grid of true Cartesian coordinates onto the space. Different colours denote respective axes the true coordinates.
that
trained twice with random initialisation. Each subplot shows the orthogonal projection of the space onto the plane spanned by 21 and 23. therefore hiding most of the y-axis (i.c. ball height) wherever linear disentanglement is fulfilled. During training, the role of physical symmetry is "straighten" the encoded grid and a larger K yields stronger effect.
5.3 Counterfactual representation augmentation with deliberately incorrect group assumptions
Additionally, test SPS with deliberately incorrect group assumptions. The motivation is as follows. In real applications, researchers may incorrectly specify the symmetry constraint when the data are complex or the symmetry is not known priori. SPS is more useful if it works with various groups assumptions close to the truth. In our analysis, we are surprised to find that SPS still learns interpretable representations under alternate group assumptions via perturbing the correct one.
Figure shows our results with the vision task (on the bouncing ball dataset). The tick labels show the augmentation method. Its syntax follows section 3.3 e.g., "(R', 1 +) )) x SO(2)" denotes augmenting representations by 1D translations and 2D rotations. The y axis of the plot is still linear projection loss (as discussed in section A.5.3) that evaluates the interpretability of the learned representation. As shown by the boxplot, five out of five perturbed group assumptions yield better results than the "w/o Symmetry" baseline. Particularly, (R3 +) SO(2) and (R2 +) x SO(3) learn significantly more linear representations, showing that some symmetry assumptions are "less incorrect" than others, and that SPS achieve good results under multitude of group assumptions.

---

0.6
Linear pro).
0.4
0.2
0.0
w/o (R", 1)
S0(2)
(R))
(R),
11
(R3) ()
(B)
Symmetty
S0(2)
S0(2)
S0(3)
S0(2)
(Ours)
Figure 8: Evaluation on various group assumptions, which shows that physical symmetry is a very robust inductive bias as even the incorrect symmetry assumptions lead to better results than baseline. Here, the y axis is linear projection loss between the learned location factor and the true coordinates, so a lower value means better interpretability of representations. The leftmost box shows the baseline without symmetry constraint. The next five boxes show five deliberately incorrect group assumptions, trained with K=4. The rightmost box shows the correct group assumption.
6 Related work
The idea of using a predictive model for better self-supervised learning has been well established
[Oord al. 2018, Chung et al.) 2015 LeCun 2022]. In terms of model architecture, our model is very similar ar to VRNN Chung et al. 2013]. In addition, our model can be seen as a variation of joint-embedding predictive architecture (JEPA) in [LeCun 2022] if we eliminate the reconstruction losses on the observation. In fact, we see the network topology of a model as the "hardware" and see the learning strategy (e-g., contrastive method, regularised method, or mixed one) the "software". The main contribution of this study lies in the learning strategy - to use physical symmetry to limit the complexity of the prior model, and use counterfactual representation augmentation to increase sample efficiency.
The existing notation of "symmetry" as in (Higgins et al. 2018 Bronstein et al. 2021 is very different from physical symmetry as an inductive for representation learning Most current symmetry- based methods care about the relation between observation and latent [Sanghi] 2020 Quessard er al.) 2020; Dupont el al.) 20200 Huang el at.) 2021]. E.g., when a certain transformation is applied v. z should simply keep invariant follow same/similar transformation. Such an assumption inevitably requires some knowledge in the domain of æ. In contrast, physical symmetry focuses solely on the dynamics of 2, and therefore we only have to make assumptions about the underlying group transformation in the latent space. We see two most relevant works in the field of reinforcement learning (Mondal al.) 2022 Dupont et al. 2020]. which apply an equivariant assumption similar to the physical symmetry used in this paper. The major differences are twofold. First, to disentangle the basic factors, our method requires no interactions with the environment. Second, our method much more concise; it needs no other tailored components or other inductive biases such as symmetric embeddings network and contrastive loss used in [Dupont et al., 2020) or MDP homomorphism applied in Mondal el al. 2022
7 Limitation
We have identified several limitations in the generality and soundness of SPS. Firstly, when the underlying concept following physical symmetry only contains partial information of the time series and cannot fully reconstruct the inputs, SPS may not function properly. We hypothesize that this issue is connected content-style disentanglement, and present some preliminary results in appendix A.4 and A.5 Secondly, the current model lacks the ability to distill concepts from multibody systems. For example, it is unable learn the concept of pitch from polyphonic music or understand 3D space from videas featuring multiple moving objects. Lastly, it is essential to develop a formalized theory for quantifying the impact of counterfactual representation augmentation in future work. This would involve measuring the degree of freedom in the latent space with and without physical symmetry, and explaining why incorrect symmetry assumptions can still result in a correct and interpretable latent space.

---

8 Conclusion
In this paper, we use physical symmetry as novel inductive bias to learn interpretable and low- dimensional representations from time-series data. Experiments show that physical symmetry effectively distills an interpretable linear pitch concept, which agrees with human music perception, from music audios without any labels. With the same method, we learn the concept of 3D Cartesian space from monocular videos of bouncing ball shot from a fixed perspective. In addition, robust training technique, counterfactual representation augmentation, developed to enforce physical symmetry during training. Analysis shows that counterfactual representation augmentation leads to higher sample efficiency and better latent-space interpretabilityy, and it stays effective when the symmetry assumption incorrect. Last but not least, we see that with physical symmetry, our sequential representation learning model can drop the the Gaussian prior regulation on the latent space. Such a result empirically indicates that physical symmetry. as causal (counterfactual) inductive bias, might be more essential compared the Gaussian prior purely statistical regularization.
Acknowledgments
We'd like thank Dr. Zhao Yang and Dr. Maigo Wang for inspiring discussion on physical symmetry. Also, we'd like to like to extend our sincere thanks to Junyan Jiang and Yixiao Zhang for their useful and practical suggestions.
References
Taketo Akama. Controlling symbolic music generation based on concept learning from domain knowledge. In Arthur Flexer, Geoffroy Peeters, Julián Urbano, and Anja Volk, editors, Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pages 816 823, 2019.
autoencoder. Advances Neural Information Processing Systems, 34: 10105-001188, 2021. Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems, 28, 2015.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: incoder-decoder approaches. In Proceedings of SSS--8 Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics.
Collins Chris. GeneralUser GS, 2017.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In C. Cortes, Lawrence, D. Lee, M. Sugiyama, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Alain De Cheveigné and Hideki Kawahara. Yin, fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111(4):1917-1990, 2002.
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning, pages 2761-2770. PMLR. 2020.
Jesse Engel, Rigel Swavely, Lamtharn Hanoi Hantrakul, Adam Roberts, and Curtis Hawthorne. Self- supervised pitch detection by inverse audio synthesis. JCML. 2020 Workshop on Self-supervision in Audio and Speech, 2020.
10

---

Philippe Esling, Axel Chemla-Romeu- Komeu-Santos, and Adrien Bitton. Bridging audio analysis, perception and synthesis with cereptually-regulaaiiin variational timbre spaces. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, pages 175-181, 2018.
Eric Foxley. Nottingham database, 2011.
Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Velimirović. Spice: Self-super pitch estimation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:11 118-1128, 2020.
Irina Higgins, David Amos, David Pfau, Sehastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.00230,, 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in Neural Information Processing Systems, 30, 2017.
Siyuan Huang. Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6535-6545. 2021.
Dasaem Jeong, Taegyun Kwon, Yoojin Kim, and Juhan Nam. Graph neural network for music score data and modeling expressive piano performance. In International Conference Machine Learning, pages 3060-3070. PMLR, 2019.
Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey on deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint arXiv:2011.0680) 2020.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021.
Yann LeCun. A path towards autonomous machine intelligence. preprint posted on openreview, 2022.
Liwei Lin, Gus Xia, Qiuqiang Kong, and Junyan Jiang. A unified model for zero-shot music source separation, transcription and synthesis. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy, editors, Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, pages 381-388, 2021.
Yin-Jyun Luo, Kat Agres, and Dorien Herremans. Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders. Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, 2019.
Yin-Jyun Luo, Kin Wai Cheuk, Tomoyasu Nakano, Masataka Goto, and Dorien Herremans. Unsuper- vised disentanglement of pitch and timbre for isolated musical instrument sounds. In Proceedings of the 21 International Society for Music Information Retrieval Conference, ISMIR 2020, pages 700-707, 2020.
Yin-Jyun Luo, Sebastian Ewert, and Simon Dixon. Towards robust unsupervised disentanglement of sequential data - a case study using music audio. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 3299-3305. International Joint Conferences on Artificial Intelligence Organization, 2022. Main Track.
Ollie McCarthy and Zohaib Ahmed, Hooligan: Robust, high quality neural vocoding. arXiv preprint arXiv:2008.02493, 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4460-4470 2019.

---

Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi. and Siamak Ravanbakhsh. Eqr: Equivariant representations for data-efficient reinforcement learning. In International Conference on Machine Learning, pages 19908-15926. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 10X.::180.03374,, 2018.
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33:19777-19737, 2020.
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3577-3586, 2017.
Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In European Conference on Computer Vision, pages 626-642 Springer, 2020.
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15182-15192. 2021.
Matthew J Vowels, Necati Cihan Camgoz, and Richard Bowden. Vdsm: Unsupervised video disentanglement with state-space modeling and deep mixtures of experts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8176-8186, 2021.
Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Julie Cumming, Jin Ha Lee, Brian McFee, Markus Schedl, Johanna Devaney, Cory McKay, Eva Zangerle, and Timothy de Reuse, editors, Proceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, pages 662-669, 2020.
Shiqi Wei and Gus Xia. Learning long-term music representations via hierarchical contextual constraints. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy, editors, Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, pages 738-745 2021.
Shiqi Wei, Gus Xia, Yixiao Zhang, Liwei Lin, and Weiguo Gao. Music phrase inpainting using long- term representation and contrastive loss. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 186-190, 2022.
Frank Wen. The fluid release general-MIDI soundfont, 2013.
Ruihan Yang, Dingsu Wang. Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. In Arthur Flexer, Geoffroy Peeters, Julián Ur- bano, and Anja Volk, editors, Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pages 596 0006600,222 2019.
Li Yi, Haochen Hu, Jingwei Zhao, and Gus Xia. Accomontage2: A complete harmonization and accompaniment arrangement system. In Ismir 2022 Hybrid Conference, 2022.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5679, PMLR, 2018
Yizhe Zhu, Martin Renqiang Min, Asim Kaday, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pages 2020.
12

---

A Appendix
The appendix is structured into three main parts.
The first part (section A.I
provides additional details about SPS. Section A. focuses on implementation--rlated aspects, while section
A.2 presents experimental results concerning the
reconstruction and prediction loss.
The second part (section t.3 A.5) introduces an extended version of SPS called SPS+. Section A.3
describes the capabilities of SPS+ in achieving content-style disentanglement along with interpretable learning. The related experiments are presented in section A.4 and section A.5
The third part (section A.66) presents two additional complex experiments conducted separately using SPS+ and SPS, respectively.
A.1
SPS implementation details
A.1.1 Architecture details
Our models for both tasks share the following architecture. The encoder first uses a 2D-CNN with ReLU activation shrink the input down
x 8 middle layer, and then a linear layer to obtain the encoder in a VAE (instead of an AE), two linear layers characterises the posterior, one for the mean and the other for the log-variance. The prior model is vanilla RNN of one layer with 256 hidden units and one linear layer projection head. The decoder consists of small fully-connected network followed by 2D transposed convolution layers mirroring the CNN in the encoder. Its output is then passed through sigmoid function. We use no batch normalisation or dropout layers.
Minor variations exist between the models for the two tasks. In the audio task, convolution layers in the encoder, with three linear and three 2D transposed convolution layers in the decoder. In the vision task, the data encoder, with four linear and four 2D transposed convolution layers in the decoder.
three
complex,
four convolution layers in the
A.1.2 Training details
For both tasks, we use the Adam optimiser with learning rate 10-3 The training batch size is 32 across all of our experiments. For all VAE-based models, including SPSvAE (ours/ablation) and 3-VAE (baseline), we set 33 (i.e., X3 in Equation (T)) to 0.01, with \ 1 and X2 2. All BCE and MSE loss functions are calculated in sum instead of mean. K for all SPS models except for those discussed in section 5 where analyse the influence of different K.
The RNN predicts 1:+ given the first n embeddings 21:n- We choose
for the audio task and 5 for the vision task. We adopt scheduled sampling (Bengio et al.) 2015) during the training stage, where we gradually reduce the guidance from teacher forcing. After around 50000 batch iterations, the RNN relies solely on the given æ1.7 and predicts auto-regressively
A.2 SPS reconstruction and prior prediction results
We investigate the reconstruction and prediction capacities of our model and show that they are not harmed by adding symmetry constraints. For the music task, we compare our model, our model ablating symmetry constraints, and a 3-VAE trained solely for the reconstruction of power spectrogram. Table reports per-pixel BCE of the reconstructed sequences from the original input frames (Self-recon) and from the RNN predictions (Image-pred), We also include Lipelor- the MSE loss on the RNN predicccd 8 as defined in section 3.2 The results show that our models slightly surpasses the ablation and baseline models in all three metrics.
Similarly, Table displays the reconstruction and prediction losses on the test set for the video task. Results show that adding symmetry constraints does not significantly hurt the prediction losses. Frame-wist self-reconee reconstruction is significantly lower for the SPS models, but only by a small margin.
A.3 SPS+
SPS can use physical symmetry to learn interpretable factors that evolve over time. We call those factors content representation. However, many problems not be represented by content represen-
13

---

Table 3: Reconstruction and prediction results on the audio task.
Methods
Self-recon
Image-pred +
Cpior
SPSS
K=4 (Ours)
0.0292:10.003
0.0296::0.0003
0.0012::0.006
SPSAF. K= (Ours)
0.0292:+0.0002
0.0296:10.0000
0.0012:10.0003
SPSvAR, K== (Ablation)
0.0090
0.0095
0.0030==0.0033
SPSAE.
K==) (Ablation)
0.0001
0.0295+10.000
0.0087=-0.0212
3-VAE
0.0303+0.0008
N/A
N/A
Table 4: Reconstruction and prediction losses of the video task. Two outliers are removed from the 50 runs.
Method
Self-recon +
Image-pred +
Sprior +
SPSvAE, K=4 (Ours)
0.64382 + 9e-05
0.6456 + 4e-04
0.14 + 0.05
SPSAE: K=4 (Ours)
0.64386 7e-05
0.6458 3e-04
0.17 0.07
SPSVAE. K=0 (Ablation)
0.64372 + 4c-05
0.6459
+ 2c-04
0.19 = 0.10
SPSAE
K=0 (Ablation)
0.64367 5e-05
0.6456 1e-04
0.11 0.03
8-VAE
0.64345 + 5e-05
N/A
N/A
tation alone. For example, the bouncing balls can have different colours and the pitch scales be generated by different instruments. the colour of a ball or the timbre of a sound scale are constant within trajectory, those latent spaces hard to constrain by physical symmetry. We call such invariant factors style representation. In order to deal with these problems, we combine SPS with simple content-style disentanglement technique: SPS+, a more general framework of SPS. We random pooling to constrain the style factors, and use physical symmetry to constrain the content representation the same way SPS in Section 3.1
A.3.1 Model
Figure shows the design SPS+, which belongs the family of disentangled sequential autoen-
coders Bai et al, 2021 Hsu et al. 2017 Vowels et al. 2021 Yingzhen and Mandi 2018 Zhu ei al. 2020). During the training process, the temporal data input X1.7 first fed into the encoder E to obtain the corresponding representation Z1.7 Z1.7 is then split into two parts: the style factor 2i:7,, and the content factor Zi:T.e. The style factor Zi is passed through the random- pooiooon--ooggggggoo module P. where one element 2r,s is randomly picked. The content factor ZiT,e is fed into three branches, then combined with =r,s to reconstruct. For random pooling the training stage, one style vector randomly selected from all time steps (i.e., 15 for the music task and 20 for the vision task) of the sequence represent 2g. In the testing stage, only the first (vision task) or 3 (music task) frames are given, and will be selected from them.
A.3.2 Training objective
The following loss functions in SPS+ slightly vary from those in SPS. For SPS+, Lprior and Laym work on the content part of latent variables only. Other loss functions are exactly the same as those defined in section 3.2
Eprior Ez(T22 Zerr,)),
(8)
Caym = (2(22. 22 + ( (2(22 22T.c)
(9)
tg (Z.zT.eer Z2:T.e)
1 K
Σ (R(S(((722) 22.,))
(10)
k=1
14

---

Z1.T,
X1.
split
Zi:T
R
22
R2
S
22.+ 1,0
X2.T+1
2 1,
Figure 9: An overview of our model. X1:T is fed into the encoder E obtain the corresponding representation žirr, which is then split into two parts: the style factor Zı:T.s and the content factor Zi:T.e. The style factor is passed through the random-pooling layer P, where an element 2r,, is randomly selected. The content factor is fed into three different branches and combined with 2r,, to reconstruct three outputs respectively: Xi.r X2:7 and N2.r-1- Here, the prior model and is the symmetric operation. The inductive bias of physical symmetry enforces R to be equivaraint w.r.t. to S, so Z and should be close to each other and so are x and X.
l2 (22:T, 22.T,e)
1
Σ
(2(5
(11)
K
K
(RCr(X2.T,XX.77
(MCS 2r,a), X2.T).
(12)
K
A.4
SPS+ on learning pitch & timber factors from audios of multiple instruments
A.4.1 Dataset and setups
synthesise a dataset that contains around 2400 audio clips played by multiple instruments. Similar to the dataset in section 4.I.1 each clip contains 15 notes in major scales with the first notes ascending and the last 8 notes descending. Each note has the same volume and duration. The interval between every two notes is equal. We vary the starting pitch such that every MIDI pitch in the range C2 to C7 is present in the dataset. For each note sequence, we synthesise it using 53 different instruments, yielding 2376 audio clips. Specifically, two soundfonts clips respectively: FluidR3 GM [Wen] [013) for the train set and General GS v1.471 [Chris] 2017] for the test set. The pitch ranges for different instruments vary, so limit each instrument to its common pitch range (See Table [4).
used render those audio
We assume 2, R and E R², and use random S E G 2
IR
to augment z. with K=4,
A.4.2 Results on pitch-timbre disentanglement
We evaluate the content-style disentanglement using factor-wise data augmentation following (Yang al. 2019]. Namely, we change (i.e., augment) the instrument (i.e., style) of notes while keeping their pitch the same, and then measure the effects on the encoded 2, and Zg. We compare the normalised Ze and 2g. ensuring they have the same dynamic range. Ideally, the change of 2 should be much more significant than 2c. Here, we compare four approaches: 1) our model (SPS+), 2) our model without splitting for 2, (SPS with E R and S G (R, +)) an ablation, 3) GMVAE [uo e al. 2019]. a domain-specific pitch-timbre disentanglement model trained with explicit pitch labels. and 4) TS-DSA [Luo et al.) 2022). recent unsupervised pitch-timbre disentanglement model based on Disentangled Sequential Autoencoder (DSAE).
Figure 10 presents the changes in normalised =, and z, measured by L.2 distance when we change the instrument of an anchor note whose pitch D3 and synthesised by accordion. 5 more quantitative version by aggregating all possible instrument combinations and all different
15

---

|
||Azal|a
SPSvAE+ (Ours)
SPSvAE (Ablation)
GMVAE (Baseline)
TS-DSAE (Baseline)
5.0
2.5
Clarinet
E. Piano
Plute
Guitar
Piano
Saxophonee
Trumpet
Violin
Clarinet
E. Piano
Plute
Guitar
Piano
Saxophone
Trumpee
Viulin
Clarinet
E. Plano
Flute
Guitar
Piano
Sazophone
Trumppt
Violin
Clarinet
E. Piano
Plute
Guisar
Piano
Saxophone
Trumpet
Violin
Figure 10: Comparisons for a and Az, for different instruments against accordion, with pitch kept constant at MIDI pitch D3. Aze and 2 are changes in normalised 2e and 2g, so that higher black bars relative to white bars means better results. All results are evaluated on the set.
Table 5: Mean ratios of changes in normalised &e and 2, under timbre augmentation across all possible instrument combinations under different constant pitches in the test set.
Methods
|| z / 222
SPS (Ours)
0.49
SPSvAF (Ablation)
2.20
GMVAE (Baseline)
0.67
TS AS-DSAE (Baseline)
0.65
pitch pairs. Both results show that SPS+ produces a smaller relative change in 2, under timbre augmentation, demonstrating a successful pitch-timbre disentanglement outperforming both the ablation and baseline. Note that for the ablation model, %e varies heavily under timbre augmentation, seemingly containing timbre information. This result indicates that the design of an invariant style factor over the temporal flow is necessary to achieve good disentanglement,
We further quantify the results in the form of augmentation- baaeed based queries following Yang er al. 2019). regarding the intended split in 2 as ground truth and the dimensions with the largest variances from factor-wise augmentation after normalisation as predictions. For example, under timbre augmentation under given pitch for our model, if 21 and 23 are the two dimensions of that produce the largest variances after normalisation, we count one false positive (21), one false negative (22), and one true positive (23). The precision would be 0.67. Table ( shows the precision scores of the four approaches against their corresponding random selection. The results are in line with our observation in the previous evaluation, with our model more likely to produce the largest changes in dimensions in &e under content augmentation and that z, under style augmentation.
Table 6: Results augmentation-basee based queries the audio task. Precision, recall and F1 the same since the number of predicted and ground-truth positives are identical. Note that random precisions for different approaches can be different as 2, and 2, are split differently.
Methods
Timbre augmentation
Pitch augmentation
Precision f Random Precision t Random
SPS (Ours)
0.98
0.67
0.82
0.33
SPS (Ablation)
0.50
0.67
0.02
0.33
GMVAE (Baseline)
0.93
0.50
0.83
0.50
TS-DSAE (Baseline)
0.81
0.50
0.68
0.50
16

---

SPSSAE K K=4 (Ours)
SPS =0 (Ablation)
8- V-V (Baseline) SPICE (Baseline)
Embeddlng
2.5
0.75
0.0
0.50
-2.5
0.25
C2
C4
C6
C2
C4
C6
C2
C4
O6
C2
C4
C6
Pitch
Pisch
Pitch
Pitch
Accordion
Guitar
Synthesis
Detected Piteh
C6
O6
C6
Sacophone
Violin
C44
C4
C4
Clarinet
Piano
Plano
Flute
C24
C2
Trumpet
-25
0.0
0.0
2.5
Ppiic
Rydid
Figure 11: A visualisation of the mapping between the 1D content factor and the true pitch. In the upper row, models encode notes in the test set to Zpitch. The x axis shows the true pitch and the y axis shows the learned pitch factor. In the lower row, the axis traverses the zpitch space. The models decode Spitch to audio clips. We apply YIN to the audio clips to detect the pitch, which is shown by the y axis. In both rows, linear, noiseless mapping is ideal, and method performs the best.
Table 7: Reconstruction and prediction results the audio task.
Methods
Self-recon + Image-prec 4 Cprior +
SPSv K=4 (Ours)
0.0356
0.0359
0.0418
SPSV K=0 (Ablation)
0.0360
0.0363
0.0486
B-VAB (Baseline)
0.0359
N/A
N/A
A.4.3 Results interpretable pitch space
Figure II shows that the pitch factor learned by SPS+ has linear relation with the true pitch. Here, we use Spitch as the synonym of z. to denote the content factor. The plot shows the mappings of two tasks and four models. In the embedding task (the first row), r-axis is the true pitch and y-axis is embedded Zpitch- In the synthesis task (the second row), x-axis is Zpitch and y-axis is the detected pitch (by YIN algorithm, standard pitch-estimation method by (De Cheveigné and Kawahara 2002) of decoded (synthesised) notes. The fours models involved are: 1) our model, 2) our model without symmetry (K=0), 3) 3-VAE trained encode single-note spectrograms from single instrument (banjo) to ID embeddings, and 4) SPICE Gieller et al.) [2020]. a SOTA unsupervised pitch estimator with strong domain knowledge on how pitch linearity reflected log-frequency spectrograms. As the figure shows, without explicit knowledge of pitch, our model learns more interpretable pitch factor than 3-VAE, and the result is comparable to SPICE.
Figure 12 shows a more quantitative analysis, using R2 as the metric to evaluate the linearity of the pitch against Zpitch mapping. Although SPICE produces rather linear mappings in Figure 11 suffers from octave errors towards extreme pitches, hurting its R?² performance.
A.4.4 Reconstruction and prior prediction
We investigate the reconstruction and prediction capacities of our model and show that they are not harmed by adding symmetry constraints. We compare our model, our model ablating symmetry constraints, and a VAE trained solely for only image reconstruction. Table 7 reports per-pixel BCE of the reconstructed sequences from the original input frames (Self-recon) and from the RNN predictions (Image-pred). We also include Lprior, the MSE loss on the RNN RNN-predicted é as redefined in section A.3.2] The results show that our model surpasses the ablation and baseline models in all three indexes.
17

---

S, K=d (Ours)
XXA
Ø-VAE (Baseline)
SPICE (Baseline)
Embedding
0.5
0.0
Synthesis
1.0
R
Piano
Accordion
Clarinet
Electric Piano
Flite
Chitar
Saxophone
Trumpet
Violin
Figure 12: We use R to evaluate mapping linearity. larger R² indicates a more interpretable latent
space. Results are evaluated on the test set.
A.5 SPS+ learning space & colour factors from videos of colourful bouncing balls
A.5.1 Dataset and setups
We run physical simulations of a bouncing ball in 3D space and generate 4096 trajectories, yielding dataset of videos. Similar the dataset in section 4.2.1 the simulated ball is affected by gravity and bouncing force (clastic force). A fixed camera records a 20-frame video of each 4-second simulation obtain one trajectory (see Figure 13). The ball's size, gravity, and proportion of energy loss per bounce are constant across all trajectories. In this dataset, the color of the ball varies by trajectory, rather than a single color. For each trajectory, the ball's ' colours are uniformly randomly sampled from continuous colour space.
Figure 13:
Two example trajectories from the bouncing ball dataset.
We set 24 E Ra with the same counterfactual representation augmentation as section 4.2 (S G = (IR (2(,), K=4). Two of its dimensions intended to span the horizontal plane and the third unaugmented latent dimension intended to encode the vertical height. We set 2, E R2 which is intended to represent the ball's colour space.
A.5.2 Result on space-colour disentanglement
Similar to section A.4.2, we evaluate the space-colour disentanglement by augmenting the colour (i.c., style) of the bouncing balls while keeping their locations, and then measure the effects the normalised ze and z., Again, good disentanglement should lead to a change in 2, much more significant than 2.. Here, compare two approaches: I) our model (SPS+) and 2) our model ablating splitting for 2, (SPS with E RD and S E G z (R2,+) x SO(2)). Note that the ablation model does not differently constrain ž2 (corresponding to the y-axis) than 2g. To ensure meaningful comparison, under colour augmentation, we consider 22 to be part of 2, of the ablation model and part of z, of the complete model.
Figure presents the changes in normalised 2, and 2, measured by distance when we change the colour of an anchor ball whose location is (0, 1, 5) and rendered using white colour. Table
18

---

Table 8: Results on augmentation based queries on the visual task. Since the ablation model does not differently constrain 22 (corresponding to the y-axis) than 2x, we consider 2G and 2, differently for the two approaches. Under colour augmentation, we consider 22 to part of 2s for the ablation model and part of for the complete model. Under location augmentation, we consider to be part of 2e for both models.
Methods
Colour augmentation Location augmentation
Precision Random
Precision
Random
SPSvÆE+ (Ours)
0.99
0.40
0.88
0.40
SPSvaE (Ablation)
0.64
0.60
0.36
0.40
Table 9: Mean ratios of changes in normalised 2e and 2, under colour augmentation across sampled colour combinations keeping locations constant. Results are evaluated on the test set.
Methods
|| 1Azel12/ /2/ |2
SPS (Ours)
0.54
SPSVAE (Ablation)
1.62
provides quantitative version by aggregating sampled colour combinations and location pairs. Both results show that our model produces a smaller relative change in &e under timbre augmentation, demonstrating successful pitch-timbre disentanglement outperforming the ablation model. Note that for the ablation model, & varies heavily under colour augmentation. Table g shows the precision scores of the SPS+ and its ablation against their corresponding random selection for the ball task. These results agree with section A.4.2 and again indicate that the design of an invariant style factor helps with disentanglement.
Azz||a
|As,1|a
SPSvAR: (Ours)
SPSvAR (Ablation)
Blue
Cyan
Green
Pink
Med
Yellow
Blue
Cyan
Green
Pink
Red
Yellow
Figure 14: Comparisons of normalised Az. and Az, for different colours against white, with the ball's location kept constant at (0, 5). Higher black bars (relative white bars) means better result. (Results are evaluated on the test set.)
Figure 15 eevaluates the learned colour factor of our model. Rach pixel shows the colour of the ball synthesised by the decoder using different z coordinates. The ball colour detected using naive saturation maxima. In the central subplot, the location factor 21.3 stays at zeros while the colour factor 24:5 is controlled by the subplot's x, y axes. As shown in the central subplot, our model (a) learns a natural 2D colour space. The surrounding subplots keep the colour factor 24:5 unchanged, and the location factor 21,3 controlled by the subplot's 's 2, y axes. A black cross marks the point where the entire 21.5 is equal to the corresponding black cross in the central subplot. As is shown by the surrounding subplots, varying the location factor does not affect the colour produced by our model (a), so the disentanglement is successful. The luminosity changes because the scene is lit by point light source, making the ball location affect the surface shadow. On the other hand, / VAE (b) leams uninterpretable colour factor.
19

---

(a) SPSSAEt, K=4 (Ours)
(b) &-AA (Baseline)
76
R3
A4
Fg
Rg
R1
25 25 R0 70
F1
Figure 15: The colour map of the synthesised ball experiment through latent space traversal. Each pixel represents the detected colour from one synthesised image the ball. Each subplot varies two dimensions of showing how the synthesised colour responds the controlled
SPSvvE hs (Ours)
(b) SPS (Ablation)
(c) A-VA (Baseline)
Figure 16: Row shows the generated images when changing and keeping žyi 0, where the axis varies 2 from -2 to +2a. In (a), changing 22 controls the ball's height, and changing 21, 23 moves the ball parallel to the ground plane.
A.5.3 Results on interpretable 3D representation
Figure 16 illustrates the interpretability of learned content factor using latent space traversal. Each row varies only one dimension of the learned 3D content factor, keeping the other two dimensions at zero. Figure a) shows the results of our model. We clearly observe that: i) increasing 21 (the first dimension of Ze mostly moves the ball from left right, increasing 22 moves the ball from bottom to top, and increasing 23 mostly moves the ball from far to near. Figure 16]b) is the ablation model without physical symmetry, and (c) shows the result of our baseline model 3 VAE, which is trained to reconstruct static images single colour (green). Neither (b) nor (c) learns an interpretable latent space.
Table 10: Linear fits between the true location and the learned location factor. We run the encoder on the test set to obtain data pairs in the form of (location factor, true coordinates). We then run linear fit on the data pairs to evaluate factor interpretability.
Method
axis MSE
y axis MSE +
axis MSE
MSE 1
SPSwaE K=4 (Ours)
0.11
0.06
0.09
0.09
SPSSAR+. K=0 (Ablation)
0.35
0.72
0.68
0.58
3- 3-VAE (Baseline)
0.37
0.76
0.73
0.62
20

---

Table 11: Reconstruction and prediction results on the video task with variable colours.
Method
Self-recon | Image-pred + Cprior +
SPSS K=4 (Ours)
0.6457
0.6464
0.0957
SPS K=O (Ablation)
0.6456
0.6464
0.1320
3-VAE (Baseline)
0.6455
N/A
N/A
Table 12: R² aggregated across all instruments the test set. interpretable latent space.
larger R
indicates a more
Method
Self-recon |
Image-pred +
Cprior V
Embedding R²
Synthesis R
SPSAÆE+. K44 (Ours)
0.0384
0.0396
0.7828
0.89
0.47
SPSyAE+. K=0 (Ablation)
0.0388
0.0400
0.9909
0.83
0.25
3-VAE (Baseline)
0.0324
N/A
N/A
0.19
0.29
Table I0 quantitatively evaluates the linearity of the learned location factor. We fit a linear regression from 2, the true 3D location over the test and then compute the Mean Square Errors (MSEs). smaller MSE indicates a better fit. All three methods (as used Figure (6) are evaluated on a single-colour (green) test set. Results show that model achieves the best linearity in the learned latent factors, which aligns with our observations in Figure [6]
A.5.4
Reconstruction and prior prediction
Similar to section A.4.4 we show that our model suffers little decrease in reconstruction and prediction performance while surpassing the ablation model in terms of Lpiirr by table [1
A.6 More complicated tasks
The main part of this paper focuses simple, straight-fowwate experiments. Still, supplement findings by reporting our current implementation's 's performance on more complicated tasks involving natural melody and real-world video data.
A.6.1 Learning interpretable pitch factors from natural melodies
We report the performance of SPS+ learning interpretable pitch factors from monophonic melodies under a more realistic setup. We utilize the melodies from the Nottingham Dataset [Foxley] 2011]. collection of 1200 American and British folk songs. For simplicity, we quantise the MIDI melodies by eighth notes, replace rests with sustains and break down sustains into individual notes. We synthesise each non-overlapping 4-bar segment with the accordion soundfonts FluidR3 GM (Wen 2013). resulting in around 5000 audio clips, each of 64 steps.
This task is more realistic than the audio task described in A.4 since we large set of natural melodies instead of one specified melody line. The task is also more challenging as the prior model has to predict long and more complex melodies. To account for this challenge, et al. 2014) with layers of 512 hidden units as the prior model. We perform early-stopping after around 9000 iterations based spectrogram reconstruction loss the training set. The model and training setup is otherwise the same as in A.4
GRU [Cho
Following A.4.3 We evaluate our approach on notes synthesised with all instruments in General GS v1.471 Chriss 2017) in the MIDI pitch range of C4 to C6, where most of the melodies in Foxley [001 take place. Note that this is challenging zero-shot scenario since the model is trained only one instrument. We compare our model, our model ablating the symmetry loss and 3 B-E baseline. We visualise the embedded Zpitch and synthesised pitches for different instruments in Figure 17 Following 12 R2 results are shown in Figure 18 and Table 12 Even when tested on unseen timbres, our model can learn linear and interpretable pitch factors demonstrates better embedding and synthesis performance compared with the ablation model, which outperforms the 3- 3-VAE baseline.
21

---

VAR f
SAET (Ablation)
(Ours)
3-VA (Baseline)
Acccedion
Guitar
Embeddding
Sacophoue
Violin
de
Cartiees
Plano
Flute
Trumpet
C4
O6
Pisch
Pitch
Pitch
46+
O6
06
Deterted Pinds
Syuthesis
Spitch
Pybck
Spoids
Figure 17: A visualisation of the mapping between the embedded ID content factor and the true pitch for the model trained on Nottingham dataset.
PS K= (Ours)
SPS K=O (Ablation)
2-VAE (Baseline)
Embedding
R
0.5
0.0
Synthesss
0.3
XXX2
0.0
Axcuediion
Piano
Clarinet
Electric Piano
Plate
Guitar
Saxophone
Trumpot
Violin
Figure 18: R² for select instruments in the test set. A larger R² indicates a more linear and interpretable latent space.
A.6.2 Learning an intepretable location factor from KITTI-Masks
In this task, we evaluate our method's capability real-world dataset, KITTI-Masks [Klindt et al. 2021]. The dataset provides three labels for each image: and Y for the mask's 2D coordinate, and ARt for the pixel-wise area of the mask. Based on the provided labels, we use simple geometrical 1/ tan(av AAR), where is
relation to estimate the person-to-camera distance d, computed constant describing the typical camera's 's Field View (FoV).
We use 3-dimensional latent code for all models. For SPS, all 3 dimensions are content factors & and no style factor 2, is used. We apply group assumption (R33+) augment representations with K = 1. To measure the interpretability, we fit a linear regression from to the ground truth labels and calculate MSES in the same way as section A.5.3 The results are shown in Table [3 Linear proj. MSE 1 measures the errors of linear regression from =, to the original dataset labels. Linear proj. MSE measures the errors of linear regression from 2, to the person's 3-D location, estimated from the labels.
22

---

Table 13: Results of KITTI-Masks task, averaging on random initialisations for each method.
Methods
Self-recon |
Image-pred V
Linear proj. MSE
Linear proj. MSE
SPS K=4 (Ours)
0.030:0.0001
0.084:100066
0.215=:00067
0.203=00.065 0.243=#0008
SPSvAE. K=0 (Ablation)
0.030=10.001
0.093=0.011
0.235=:0.077
B-VAE (Baseline)
0.028=:0.001
N/A
0.403::.1994
0.399:10.204
As is shown in Table 13 MSE is smaller than MSE 1 for SPS, indicating that SPS learns more fundamental factors (person's 's location) rather than superficial features (pixel-wise location and area). For the baseline methods, MSE is almost equal to MSE and both them higher than those of SPS. In summary, our experiment shows that SPS learns more interpretable representations than the baseline (as well as the ablation method) on KITTI-Masks dataset.
(a) SPSvAE, K= (Ours)
(b) SPSvAE, K= (Ablation)
(c) B B-VE (Baseline)
21
72
2.0
0.0
2.0
-2.0
0.0
2.0
-2.0
0.0
2.0
Figure 19: Latent space traversal on diffeernnt models. Row shows the generated images when changing 2; and keeping 25. 0. For our model, the range of 21 from -2 to corresponds to the human location from near-right far-left, 22 from near-left to far-right, and from near to far. We can see that other methods produce more non-linear trajectories, for example (c), the human location hardly changes when 21 0, but it changes dramatically when 0.
Figure 19 shows the generated images, which illustrates that the factors learned by SPS are more linear than those learned by other methods in the human location attribute. For the experiment, We choose all sequences with length > 12 from KITTI-Masks our dataset; use 1058 sequences for training and 320 sequences for evaluation; In the inference stage, only the first frames are given. All three methods are trained 30 times with different random initializations. Table 13 shows the average results evaluated on the same test set with 30 different seeds.
23

---

MIDI Note
MIDI Note
Instrument
(from)
(to)
Accordion
58
96
Acoustic Bass
48
96
Banjo
36
96
Baritone Saxophone
36
72
Bassoon
36
84
Celesta
36
96
Church Bells
36
96
Clarinet
41
84
Clavichord
36
84
Dulcimer
36
84
Electric Bass
40
84
Electric Guitar
36
96
Electric Organ
36
96
Electric Piano
36
96
English Horn
36
85
Flute
48
96
Fretless Bass
36
84
Glockenspiel
36
96
Guitar
36
96
Harmonica
36
96
Harp
36
96
Harpsichord
36
96
Horn
36
96
Kalimba
36
96
Koto
36
96
Mandolin
36
96
Marimba
36
96
Oboe
36
96
Ocarina
36
96
Organ
36
96
Pan Flute
36
96
Piano
36
96
Piccolo
48
96
Recorder
36
96
Reed Organ
36
96
Sampler
36
96
Saxophone
36
84
Shakuhachi
36
96
Shamisen
36
96
Shehnai
36
96
Sitar
36
96
Soprano Saxophone
36
96
Steel Drum
36
96
Timpani
36
96
Trombone
36
96
Trumpet
36
96
Tuba
36
72
Vibraphone
36
96
Viola
96
Violin
36
96
Violoncello
36
96
Whistle
48
96
Xylophone
36
96
Table 14: Pitch range (in MIDI note) for each instrument our dataset.


Reviews:
[{"review_id": "KUSQaElmJ5", "reviewer": "Reviewer_wTJe", "summary": "This study introduces a new methodology to learn interpretable representations from data by incorporating physical symmetry as a self-consistency constraint in the latent space. It addresses a key challenge in machine learning: how to learn interpretable low-dimensional factors from unlabeled data without relying on domain-specific knowledge. The method, named Self-Supervised learning with Physical Symmetry (SPS), uses the concept of physical symmetry to ensure that the model's learned dynamics are invariant under certain transformations.\n\nThe research applies SPS in two domains: music and computer vision. In music, SPS successfully learns a linear pitch factor from unlabeled monophonic audio, and in computer vision, it learns a 3D Cartesian space from unlabeled videos of a simple moving object. The study suggests that the use of physical symmetry could lead to a new technique called representation augmentation, which enhances the model's sample efficiency.\n\nThe authors draw inspiration from the approach in modern physics where scientists often start from a symmetry assumption to derive laws and predict properties of fundamental particles. This idea of symmetry as a fundamental guiding principle is being used as an \"inductive bias\" for their representation learning model, helping to create an interpretable low-dimensional latent space.\n\nThey view symmetry in physical law not only as a design principle of nature, but also as a cognitive bias of the human mind. This suggests that they believe the learning model should reflect the same intuitive understanding of symmetry.\n\nPhysical symmetry is also used as the basis for a learning technique called \"representation augmentation\". This involves generating additional pairs of training samples from existing ones by applying certain group transformations, which are informed by the symmetries. This process helps to improve sample efficiency and imposes a regularization effect on the learning model.\n\nFinally, through the lens of causality, the authors describe physical symmetry as a counterfactual inductive bias. This means the learning model is designed to ask \"what if\" questions, specifically what would happen if predictions were based on transformed latent codes. This concept also provides constraints on the model's encoder and decoder components, since they are trained end-to-end, keeping the overall structure and function of the model consistent with the underlying symmetry principle.\n\nKey Objectives:\n\nTo introduce and explore the use of physical symmetry as a self-consistency constraint in the latent space of time-series data.\n\nTo develop the SPS methodology that applies physical symmetry to the prior model in an encoder-decoder framework.\n\nTo demonstrate the application of SPS in learning a linear pitch factor from unlabeled monophonic music audio, without any domain-specific knowledge about pitch scales.\n\nTo apply the SPS methodology in the computer vision domain, specifically learning a 3D Cartesian space from unlabeled videos of a simple moving object.\n\nTo examine the desirable properties of SPS, including conciseness, sample efficiency, robustness, and extendability.\n\nTo prove that even with an incorrect symmetry assumption, SPS can still learn more interpretable representations than baseline models.\n\nTo demonstrate the possibility of combining SPS with other learning techniques, allowing for content-style disentanglement from temporal signals.", "questions": "How do the authors propose this approach would be scalable for other tasks and what are the ways to prevent overfitting with the approach? \n\nSPS can still learn interpretable representations even with incorrect symmetry assumptions. What might be the theoretical basis for this? Could there be instances where this doesn't hold true?\n\nHow does the proposed SPS model relate to, or differ from, other models or techniques used in representation learning? Could SPS be integrated with other methods to address its limitations?", "limitations": "Authors do inform about the range of limitations for the work \n\nPartial Information Capture: The authors note that when the underlying concept following physical symmetry only encompasses a part of the information present in the time series data and cannot fully reconstruct the inputs, SPS may not work effectively. This issue is potentially related to content-style disentanglement.\n\nInability to Distill Concepts from Multibody Systems: The current model struggles with learning concepts from systems with multiple interacting components. As examples, the authors mention that SPS fails to learn the concept of pitch from polyphonic music, or understand 3D space from videos featuring multiple moving objects.\n\nLack of Formal Theory for Quantification: The authors highlight the need for a formalized theory to quantify the impact of representation augmentation. Such a theory could help measure the degree of freedom in the latent space both with and without physical symmetry, and help explain why even incorrect symmetry assumptions can still lead to correct and interpretable latent space.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Inductive Bias - By leveraging physical symmetry as an inductive bias, the learning model aims to uncover meaningful and interpretable low-dimensional representations from high-dimensional data. This approach aids the model in abstracting critical features of the data, such as the pitch of music notes or the 3D location of a moving object, while filtering out less relevant information.\n\nRegularization and Model Robustness - The method uses symmetry-based representation augmentation and transformations, which generate additional training samples. This augmentation process improves sample efficiency and imposes regularization on the model, reducing overfitting. It also helps in maintaining consistency in predictions, thereby improving model robustness.\n\nThe model is designed with self-consistency in mind. The prior model's predictions for a transformed version of the latent space and the original version should be close, and so should their decoded reconstructions. This self-consistency constraint enhances the regularization of the latent space.\n\nTraining Objectives - The total loss function used in the model is a combination of several loss terms including reconstruction loss, prior prediction loss, symmetry-based loss, and KL divergence loss. This diversified loss function enables the model to optimize different aspects of the learning process, contributing to a better overall model performance.\n\nFlexibility and Generality - This approach is versatile and can be applied to different problems. Different group transformations are used for different problems, demonstrating the method's ability to adapt to different contexts.\n\nVariants - The model has different versions like SPSVAE and SPSAE, providing flexibility based on the specific needs of the task at hand. For instance, SPSAE could be used when a simpler model architecture without the KL divergence term is sufficient.", "weaknesses": "Dependence on Correct Symmetry Assumptions: While the experiments show that the method is robust to incorrect symmetry assumptions, choosing the right symmetry transformation S is crucial for optimal results. Incorrect assumptions can still lead to an inferior performance. Real-world data may contain complex symmetries not known a priori, which could lead to difficulty in correctly specifying these assumptions.\n\nDependence on Augmentation Factor: The efficiency and effectiveness of the model are significantly affected by the augmentation factor K. Choosing an optimal K is a challenge and requires extensive experiments. A poorly chosen K could result in less efficient training or lower performance.\n\nHigh-Dimensional Latent Space: While the method has been demonstrated on 1D and 3D latent spaces, its effectiveness in higher-dimensional latent spaces is not fully explored. Real-world data can often have high-dimensional latent spaces, and it may be challenging to maintain the same level of performance in these cases.\n\nOverfitting: While representation augmentation can increase the robustness of the model, applying it excessively might lead to overfitting, especially when data is scarce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We sincerely appreciate your comprehensive understanding of this paper’s vision and strengths as well as its current limitations. Additionally, thank you for the in-depth questions. We’d like to address them here to the best of our knowledge. \n\n> **How do the authors propose this approach would be scalable for other tasks** \n\n**Reply 1**:  \nIf a dynamic system has a low-dim intrinsic latent space (similar to the 1D pitch or 3D location tasks), we expect SPS to generalise well. For tasks with *high-dim* latent factors, as the review has pointed out, applying SPS may be challenging. Similar to modern physics, trying multiple symmetry hypotheses becomes the first step. For that, we foresee an “auto-SPS” in the future which automatically *searches* for symmetry hypotheses given the search scope. Since figure 8 shows that a range of symmetry assumptions may all lead to interpretable representations, such a search has a higher probability to hit relevant symmetry assumptions. \n\nAs for tasks where physical symmetry alone is not strong enough to regularise concepts, we will need to combine SPS with other methods. We already have a simple example in our appendix: SPS+, which takes advantage of the temporally invariant style factor to enable SPS’s effect on the content factors. Alternatively, if the style factors are intractable and we do not aim to model the style factors’ temporal dynamics, we can use SPS with VICReg [1] to obtain more interpretable content factors, and preliminary results suggest that such combination outperforms either SPS or VICReg alone on some non-trivial cases. \n\n>**What are the ways to prevent overfitting with the approach?** \n\n**Reply 2**:  \nThank you for raising such a profound question. For SPS, overfitting may be caused jointly by insufficient data and the symmetry assumption. With a limited amount of data, there may be multiple ways of describing the system dynamics, and all of them may conform to the assumed symmetry, but some yield more natural representations than others. We encountered such a situation before. In the bouncing ball experiment, if all trajectories in the dataset overlap at a single point, SPS will treat that point as the origin and learn polar coordinates to represent the ground plane instead of Cartesian coordinates. Given the special dataset, polar coordinates fully adheres to the prescribed symmetry constraint. If the goal is to learn a unique interpretable representation without obtaining more data domains, the only way may be to add new symmetry constraints. However, interpretable representations are not unique to begin with. Even if the model doesn’t yield the expected representations, it has \"discovered\" new interpretable representations (e.g. polar coordinates), which can still be valuable. Analogously, if relativity represents the true system dynamics, then the Newtonian mechanics essentially overfits to macroscopic low-speed phenomena, yet it yields interpretable representations. \n\n> **SPS can still learn interpretable representations even with incorrect symmetry assumptions. What might be the theoretical basis for this? Could there be instances where this doesn't hold true?** \n\n**Reply 3**:  \nThat is an important question and we thank the reviewer for emphasising it. Our preliminary ideas include using measurement theory to characterise learnable representation dynamics with and without SPS. The correct symmetry would be the strictest equivariance constraint that is physically true. Other constraints are either unphysical or loose. Therefore, the theory that predicts the effect of incorrect symmetry assumptions can very possibly be the same theory that explains the efficacy of SPS itself, showing exactly how SPS constrains the representation dynamics. However, for a full theory please expect future work. It will be much appreciated if you would point us some directions during the discussion phase! \n\nWhile the rigorous mathematical theory behind the efficacy of SPS remains an open question, we can still consider SPS training with incorrect symmetry as a general regularisation problem in machine learning. A toy analogy is ridge regression – we know it leads to better results if the true prior is Gaussian. However, if we assume an incorrect mean, variance, penalty weight, or even to use lasso instead of L2 norm, it can still bring some benefits as long as the incorrect assumption is not “too far” from the true one. Again, to quantify “too far”, we will need a formal theory, and in Figure 8, we consider the cases to be not “too far”. \n\n> **How does the proposed SPS model relate to, or differ from, other models or techniques used in representation learning? Could SPS be integrated with other methods to address its limitations?**  \n\n**Reply 4**:  \nFrom a energy-based model point of view, SPS is a regularised, non-contrastive method. It is not an architectural method, making it compatible with other techniques. It is not a contrastive method since SPS doesn’t need contrastive samples. It is highly related to VICReg [1]; both try to find a general regularised inductive bias for the latent space, but VICReg maximises information content of predictable representations while SPS purely considers the dynamics of z time-series. \n\nSince representation augmentation is generally agonistic to the specific training pipeline, SPS can be integrated with other methods to address its limitations. In the paper we already used SPS in conjunction with VAE. In our later experiments, we try to solve an “unconstrained style factor problem” (related to A.3) by contrastive training with SPS+VICReg. VICReg is in charge of preventing representation collapse without any decoder and SPS regularises the content factor. \n\n[1] Bardes, A., Ponce, J., & LeCun, Y. (2021). Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906. \n"}, {"review_id": "B57kdkjmyt", "reviewer": "Reviewer_5wLb", "summary": "This paper proposes a symmetry equivariance constraints on the transition dynamics of latent representations for time-indexed data. The claim is that imposing these certain symmetries create interpretable model representations that correspond to popular domain-specific representations in the audio and video domains.\n\nExperiments on audio demonstrate how enforcing a translational symmetry recovers an interpretable one-dimensional latent factor corresponding to pitch. Experiments on video show how enforcing translational and rotational symmetries recovers interpretable three-dimensional latent factors corresponding to spatial coordinates. Ablations also show some robustness of learning to mis-specification of symmetries.", "questions": "Is it possible to frame the training objective (Equation 1) as a proper probabilistic loss? See [1] (the static case) and [2] (the time-indexed case) for the probabilistic formulations of the Gaussian prior. If this is possible, it might help to clarify why/whether it is possible to drop the KL regularization term.\n\nDoes this work relate to previous work on imposing symmetries on intermediate layers of neural networks? E.g. [3] and [4]. I am not deeply familiar with that line of work, but I'm bringing it up because it might be relevant (and at the very least, you might find it interesting if you aren't already aware).\n\n[1] Auto-Encoding Variational Bayes. Diederik P. Kingma, Max Welling.\n\n[2] A Recurrent Latent Variable Model for Sequential Data. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio.\n\n[3] Deep Symmetry Networks. Robert Gens, Pedro Domingos.\n\n[4] Group Equivariant Convolutional Networks. Taco S. Cohen, Max Welling.", "limitations": "The experiments presented in this work use low-dimensional latent spaces: 1 dimension for audio and 3 dimensions for video. It is not clear how to adapt these methods to the high-dimensional latent spaces commonly used (and often required) for expressive models where model performance and capacity is prioritized more highly than interpretability. Furthermore, it is not clear what symmetries we ought to impose upon high-dimensional latent spaces. That said, these questions are clearly beyond the scope of the present work: I see this limitation more as an interesting avenues for future investigation rather than a weakness of the present work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is an inspiring paper! The paper is well-motivated and well-written. The methodology is clearly described and easy to follow (although I do think there might be an even better probabilistic formulation/presentation of the training objective; see the Questions section). The methods make sense and I feel confident that I could implement these ideas myself, based on the description in the paper.\n\nThe experiments are well-executed and support the hypothesis that physical symmetries can provide a powerful inductive bias, at a higher level of abstraction than more domain-specific approaches.", "weaknesses": "My only major criticism is that the experiments are in somewhat \"toy\" settings. It would be interesting to see an application of these ideas to a more significant problem, with stronger baselines.\n\nA very minor criticism of the (otherwise excellent!) introduction: \"Such an approach is very different from human learning; even without formal music training, one can at least perceive pitch, a fundamental music concept, from the experience of listening to music.\" I think this claim is too strong. It is not clear how much of human perception is learned from experience, vs. baked in to our genetics and brain structure. This claim isn't important to your central argument, so I suggest moderating it a bit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving the insightful feedback and raising such important questions. Below we respond in a breakdown format. \n\n> **Is it possible to frame the training objective (Equation 1) as a proper probabilistic loss? See [1] (the static case) and [2] (the time-indexed case) for the probabilistic formulations of the Gaussian prior. If this is possible, it might help to clarify why/whether it is possible to drop the KL regularization term.  \n[1] Auto-Encoding Variational Bayes. Diederik P. Kingma, Max Welling.  \n[2] A Recurrent Latent Variable Model for Sequential Data. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio.**  \n\n**Reply 1**:  \nThank you for referencing [1] [2] to help us in obtaining a sound probabilistic formulation for SPS. We have been working on the theoretical grounding for SPS, but to be very frank there are no concrete results yet. We’d like to share our current direction: First, assume the dataset follows a certain symmetry. Then, show that including the representation augmentation terms in the training objective is a way of imposing a prior and maximising the likelihood of the dataset. Specifically, given the symmetry assumptions, rewrite the likelihood of the dataset to include a lower bound that is maximised by representation augmentation training (following the assumed symmetry). We will greatly appreciate your feedback in this regard! We look forward to the discussion phase and consider it as an opportunity to collaboratively discover a probabilistic formulation, in which case we will eventually and duly acknowledge your input. \n\n> **Does this work relate to previous work on imposing symmetries on intermediate layers of neural networks? E.g. [3] and [4]. I am not deeply familiar with that line of work, but I'm bringing it up because it might be relevant (and at the very least, you might find it interesting if you aren't already aware).  \n[3] Deep Symmetry Networks. Robert Gens, Pedro Domingos.  \n[4] Group Equivariant Convolutional Networks. Taco S. Cohen, Max Welling.**  \n\n**Reply 2**:  \nThank you for referencing [3] [4] as they are very exciting to read and indeed very related to learning with symmetry. [3] [4] are in the same line with [5] [6] (referenced in the paper); they emphasise some equivariant relations **between signal x and representation z**: When a certain transformation is applied to x, z should keep invariant or follow a similar/same transformation. Specifically, [3] [4] develop network architectures invariant to continuous group transformations of input data and intermediate feature maps. On the contrary, SPS **deals with z only**, and its time-series equivariance. Our paper’s major contribution is to “escape” the signal x space and only prescribe constraints in the representation z space. \n\nThere’s actually another reason we appreciate your references so much: [3]’s review on Lie groups and object orbits (section 2) gave us new insights to our method. [3] is concerned with classification tasks, characterised by their VC dimension and sample complexity, which they show can be reduced by using symmetry if object orbits are homogeneously labelled. That is analogous to our method if we consider energy-based models (EBMs) of time series. An EBM outputs an energy (scaler) given one trajectory z[1:T] which, intuitively, denotes how “physically realistic” the EBM assesses the trajectory to be. This way, the energy forms object orbits under specific symmetry assumptions, and we require each object orbit to have homogenous energy. This analogy may eventually enable new theoretical groundings for our formulation of SPS. As such we are extra thankful that you referenced [3] [4]. \n\n[5] Dupont, E., Martin, M. B., Colburn, A., Sankar, A., Susskind, J., & Shan, Q. (2020, November). Equivariant neural rendering. In International Conference on Machine Learning (pp. 2761-2770). PMLR.  \n[6] Sanghi, A. (2020). Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16 (pp. 626-642). Springer International Publishing.  \n\n> **A very minor criticism of the (otherwise excellent!) introduction: \"Such an approach is very different from human learning; even without formal music training, one can at least perceive pitch, a fundamental music concept, from the experience of listening to music.\"I think this claim is too strong. It is not clear how much of human perception is learned from experience, vs. baked in to our genetics and brain structure. This claim isn't important to your central argument, so I suggest moderating it a bit.**  \n\n**Reply 3**:  \nThank you for your kind advice. We recognize that such a strong claim calls for supportive evidence, but the psychological experiments that would prove/disprove our claim are not there yet to our knowledge. We will consider removing/soften the claim for the sake of soundness — thank you for bringing it up. "}, {"review_id": "fq1UCdxQJT", "reviewer": "Reviewer_dLV3", "summary": "This paper utilizes the concept of physical symmetry as a self-supervised constraint within an auto-encoder framework to enhance the learning of interpretable and disentangled representations. The authors validate their approach through experiments conducted on unlabelled monophonic music audio and monocular videos featuring a bouncing ball. The results demonstrate high projection accuracy and successful disentanglement of style and content. Additionally, the paper introduces representation augmentation techniques that improve sample efficiency and enhance interpretability. ", "questions": "1. Could you explain what a random-pooling layer does and the details of how you split the style and content vector? Have you ever tried other methods to disentangle the style and content part?\n2. Will the sequence length (the value of T) affect the final result?\n3. Could you describe any unique aspects or challenges of the dataset you have proposed?", "limitations": "1. The generalizability and applicability for real-world dataset may be limited.\n2. The proposed dataset should be more challenging for further utilization on research", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and easy to understand, demonstrating a cohesive and structured presentation of the research.\n2. Convincing experiment results, which align with the stated objectives, provide strong support for the findings of this study.\n3. The representation augmentation technique is novel and supported by empirical evidence showcasing its effectiveness.\n4. The results on color-texture disentanglement is clear and interpretable", "weaknesses": "1. The experimental setup seems to be somewhat simplistic despite the experimental results being good\n2. Only few experiments on real-world dataset are provided, which is not able to fully demonstrate the generalizability and applicability of your method.\n3. There is a lack of theoretical evidence of your proposed method and further explanation of the relationship between physical symmetry and representation learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review! We really appreciate the concise summarisation on both the strengths and the current limitation of the paper. We hereby respond in a breakdown format. \n\n> **Could you explain what a random-pooling layer does and the details of how you split the style and content vector? Have you ever tried other methods to disentangle the style and content part?**  \n\n**Reply 1**:  \nRandom pooling simply means we randomly pick a vector $z_{i, s}$ as $z_{\\tau, s}$ from the style factor sequence $\\textbf{z}_{1:T, s}$.  This operation encourages the learned style representation to be constant over time. Also, To split the style and content vector we first cut the vector $z$ into two parts. If $z$ has $n$ dimensions, we choose $z[0:m]$ as $z_c$ and $z[m:n]$ as $z_s$, where $n$ and $m$ are hyperparameters (For the audio problem $m=1$ and $n=2$. For the video problem $m=3$ and $n=2$).\n\nAs for other methods of disentanglement, we haven’t tried any since a random pooling (our first try) simply works within an SPS+ framework. We certainly note that the content-style definition imposed by SPS+ is very simple — content refers to representations that change over time and adhere to physical symmetries, while style refers to representations within the same sequence that remain unchanged over time (global invariance). In general, there can be meaningful factors that change over time but whose patterns of change do not adhere to physical symmetries. In such cases, we propose to combine SPS with other methods for decoupling in the future. For instance, applying SPS to Causal Variational Autoencoders (CVAEs), disentangled GANs, or contrastive/non-contrastive methods. Actually, an ongoing experiment of our team is to combine VICReg [1] and SPS to obtain more interpretable content factors, and preliminary results suggest that such combination outperforms either SPS or VICReg alone on some non-trivial cases. \n\n[1] Bardes, A., Ponce, J., & LeCun, Y. (2021). Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906.\n\n> **Will the sequence length (the value of T) affect the final result?**  \n\n**Reply 2**:  \nThe sequence length T has no direct effect on learning interpretable factors, but has an indirect effect. A sufficiently long sequence can ensure that the prior model (RNN) learns meaningful system dynamics, and this is the premise for physical symmetry constraints to work. If the prior model can already learn the correct system dynamics with a small T, increasing the sequence length will no longer help learn interpretable representations. \n\n> **Could you describe any unique aspects or challenges of the dataset you have proposed?**  \n\n**Reply 3**:  \nDue to the small size of our dataset, it is challenging for existing unsupervised learning methods to learn meaningful interpretable representations.\nThe audio dataset (as described in A.4.1) contains *only* 2400 audio clips played by multiple instruments. In the computer music domain, we know how hard it is to unsupervisedly 1) disentangle pitch and timber, and 2) learn a linear pitch concept. As far as we know, all prior works incorporate strong domain knowledge, such as pitch shifting for pseudo-label generation [1, 2] or using instrument labels [3]. \n\nLikewise, learning concepts such as 3D coordinates from unlabelled video has long been a far-fetched fantasy for CV researchers. To be fair, in recent years we have seen exciting progress on this front. For example, LEAP [4] performs physically meaningful representation disentanglement via causal discovery. But even LEAP *requires* not only independent noises but also a sufficient causal structure (e.g., five or more balls interacting in the same scene via springs forces) in order to learn disentangled location factors. \n\n[1] Luo, Y. J., Cheuk, K. W., Nakano, T., Goto, M., & Herremans, D. (2020, October). Unsupervised Disentanglement of Pitch and Timbre for Isolated Musical Instrument Sounds. In ISMIR (pp. 700-707).  \n[2] Gfeller, B., Frank, C., Roblek, D., Sharifi, M., Tagliasacchi, M., & Velimirović, M. (2020). SPICE: Self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28, 1118-1128.  \n[3] Luo, Y. J., Agres, K., & Herremans, D. (2019). Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders. arXiv preprint arXiv:1906.08152.  \n[4] Yao, W., Sun, Y., Ho, A., Sun, C., & Zhang, K. (2021). Learning Temporally Causal Latent Processes from General Temporal Data. arXiv preprint arXiv:2110.05428.  \n"}, {"review_id": "NU7QWYeN01", "reviewer": "Reviewer_mck6", "summary": "The paper presents a new way of training a self-supervised system that includes a data augmentation module in the latent space that leverages physical symmetry. This introduction of a transformation covariance in the latent space while training helps learn interpretable, robust and data-efficient representation. An evaluation is performed on synthetic data in two domains: music and videos. The evaluation shows that the new training system is able to learn a presentation that fits human perception.", "questions": "I really don't get why physical symmetry could be seen as a counterfactual inductive bias. Any latent representation model could be used to answer \"what if\" questions about the model (or rather the latent representation itself), which doesn't make them counterfactual inductive.\nCould the author be more specific about what they mean here?", "limitations": "1) The link between the group of symmetry to be used for training and the actual task may not be straightforward for tasks a bit more elaborated than the ones proposed in the paper.\n\n2) The claimed interpretability and the link with causality are questionable:\n\n   a) Sample efficiency and extendability are very barely linked to interpretability\n\n   b) The \"interpretability\" of the latent representation in the experimental part is largely due to design choices on the latent space and the symmetry, which can be considered as domain-knowledge priors. So I don't think the model helps generate interpretable representations but is only a way of encoding domain-knowledge priors. For instance, when the authors claim, \"our model learns a more interpretable pitch than...\", the model is actually not learning a more interpretable representation, but rather the prior was correctly handled by the symmetry. Same when they claim \"... leads to a better enforcement of interpretability\". It would be probably safer to talk about \"better enforcement of the prior\".\nThe example with the video has similar issues: implicitly, the chosen symmetry considers that the complicated dimension (height), which as a non-linear behavior, should be kept free, while the two others can be easily contained. This really sounds like a domain knowledge prior.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The paper is very well written and very interesting\n2) The idea of leveraging symmetry by enforcing latent space covariance seems quite novel to me and could have an impact in the field of Self-Supervised learning.\n3) The method seems to perform as claimed on the presented evaluation (on toy datasets).", "weaknesses": "1) The evaluation is done only on toy problems with synthetic datasets.\n2) Part of the evaluation is not very convincing for several reasons:\n   a) First experiment:\n- The task is too easy for the reported metrics: the R2 scores are saturated to 1, which makes the comparison with the baseline (SPICE) quite questionable.\n- The ablation study (K=0) is a bit disappointing, as the improvement of using symmetry (K=4) doesn't seem significant. So the good properties of the learned representation seem more linked to the encoder/decoder architecture than to the use of symmetry covariance.\n   b) sample efficiency: the correlation between K and sample efficiency is quite unclear in figure 6: for 512 and 2048 samples, K=4 and K=16 seem to have very similar performances.\n3) It's not trivial how the dimension of the latent space and the symmetry should be chosen for problems a bit more complex than the toy ones presented in the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing the feedback and raising important questions and concerns. We answer these questions here and also like to take the opportunity to address some concerns raised in weaknesses/limitations. \n\n> **Why could physical symmetry be seen as a counterfactual inductive bias?**  \n\n**Reply 1**:  \nWe consider the physical symmetry as a counterfactual inductive bias on latent dynamics. In other words, it addresses the question if z_[1:t] is transformed, what z_[t+1:k] would be. Other approaches, like Variational Autoencoders (VAEs), do not consider the temporal aspect of the latent z. Therefore, we do not consider them to be counterfactual inductive biases. Thank you for bringing up this issue, which made us aware of the lack of clarity in our description. We will make sure to clarify this in the revised version.\n\n> **The choices on the latent space and the symmetry group assumptions sound like domain knowledge?**  \n\n**Reply 2**  \nWe distinguish the difference between symmetry assumptions and domain knowledge through the following three aspects.\n\n**1) Representation augmentation requires much less knowledge than data augmentation.**  \nConsider pitch extraction. Almost all related works use pitch-shift augmentation, but on the data and not on the representation. It requires the knowledge of how to transform the input data to shift the pitch. Simple techniques include time stretching and spectrogram frequency stretching, and advanced techniques (vocoders etc.) involve shifting harmonics without changing the timbre. In comparison, SPS just translates the representation vector. \n\nAs for the vision task, it is not clear at all how to apply 3D spatial translation-rotation augmentation to unlabelled video. With our SPS method, one only needs to translate and rotate a representation vector of length 3. \n\n**2) Domain knowledge and representation symmetry belongs to two different conceptual levels.**  \nWhen we pick group assumptions for a given problem, the kind of knowledge required is more abstract (and general) than what we usually call “domain knowledge”. It is us human’s beliefs about concepts themselves, not how concepts are encoded from data. \n\nConcretely, recall that music and vision are two vastly different fields. Their respective domain knowledge systems look nothing like each other. In comparison, SPS solves one task with 1D translation and the other with 2D translation-rotation. Their forms look overarchingly similar! To generalise, drastically different systems can share formally similar physical symmetries. The “expertise” involved in picking symmetries is utterly different from what “domain expertise” refers to. \n\n**3) We may not even need a correct symmetry assumption for successful representation learning with SPS.**  \nThe intuition section of our paper mentions that many modern physicists *start* with symmetry (in law) assumptions *and then* obtain testable theories of physical laws. Notice: first there are guesses about symmetry, and after that, “domain knowledge” is created. \n\nWhat it means is that symmetry is so fundamentally general that proposing symmetry assumptions is an efficient way of regularising concepts, even before we settle down with any domain knowledge. One is free to try multiple symmetry assumptions and see which ones learn better representations. \n\nOur experiments (section 5.3) already show that even with *incorrect symmetry assumptions* SPS can still learn to extract 3D Cartesian coordinates from the unlabelled bouncing ball dataset. \n\n> **Part of the evaluation is not very convincing，R2 scores are saturated to 1, which makes the comparison with the baseline (SPICE) quite questionable. The ablation study (K=0) is a bit disappointing, as the improvement of using symmetry (K=4) doesn't seem significant. So the good properties of the learned representation seem more linked to the encoder/decoder architecture than to the use of symmetry covariance.**  \n\n**Reply 3**: \nThe $R^2$ score may be a bit misleading if we only focus on Table 1. If we take a comprehensive view of the experimental results, i.e. considering Table 1 alongside Figure 12, it becomes evident that the introduction of timbre factors makes the $R^2$ improvement from physical symmetry even more pronounced. As the task difficulty increases, the role of symmetry becomes increasingly significant.\n\n> **sample efficiency: the correlation between K and sample efficiency is quite unclear in figure 6: for 512 and 2048 samples, K=4 and K=16 seem to have very similar performances.**  \n\n**Reply 4**: \nWe are not trying to show a linear relation between K and interpretability (linear project loss). The point is that using representation augmentation is better than not using it, regardless of the dataset size. Additionally, it can be observed that increasing the amount of data makes it easier for the model to learn interpretable representations, so the interpretability provided by representation augmentation saturates when the data size reaches 2048. Furthermore, we are unable to determine the optimal value of K for different dataset sizes. We will consider revising the original text to clearly highlight this point, thank you for bringing it up.\n"}, {"review_id": "LJmhZM51Dy", "reviewer": "Reviewer_TYBC", "summary": "This paper presents an approach to interpretable representation learning based on ''physical symmetry''. The core idea is to learn to predict the temporal evolution of a latent variable which is additionally encouraged to be equivariant under some transformation (e.g. translation or rotation). An autoencoder-like model is designed, where an input sequence is encoded to a sequence of latent variables. From these, three objectives are defined: direct reconstruction, reconstruction after predicting the next state, and an objective involving the symmetric transform. Experiments are presented on an audio-related task (predicting pitches from a sequence of single-note input spectrograms) and a vision task (predicting the coordinates of a moving ball). Further experiments are performed to demonstrate a reduced need for training examples, interpretability of the learned representations, and style-content disentanglement.", "questions": "- How important is L_rec, relative to L_prior? That is, can we turn off L_rec and still get similar results?\n- In Equation 2 (and other equations), how do you use the binary cross entropy loss to compare spectrograms (or pictures)? This is very unusual, since spectrograms do not have a value range [0,1]. Is this applied in an element-wise fashion? \n- Line 122: ''In other words, we add or subtract the latent codes by a random scalar.'' -- Unclear. Do you mean you add a scalar *to* the latent code?\n- Line 137f: Is the choice of sequence (a scale upward and downward) relevant? Could you also choose a stationary sequence? Or a random sequence (transposed to different keys)?\n- Lines 286ff: I think the limited usefulness of the VAE compared to the AE stems from the fact that the datasets used are so simple. There are simply no sources of variability that would need to be modeled by the Gaussian prior.", "limitations": "The limitations of the synthetic experimental setups should be discussed more openly in the text.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main idea at the core of this work is interesting and original. This idea might also stimulate future work using such physical symmetry constraints. \n\nThe paper provides a variety of experiments, which seem sensibly designed. It is welcome to see experiments dedicated to two different domains (vision and audio). I especially liked the experiments on style-content disentanglement in the Appendix (SPS+). These contain much more realistic scenarios than the main text (albeit still on synthetic datasets).\n\nThe writing in the later sections of the paper is mostly clear and the figures throughout the manuscript are well done. More on the writing below.\n\nThe proposed approach shows only slight or moderate improvements compared to using no physical symmetry constraints. Nevertheless, the idea itself could be considered significant enough to be shared with the community.", "weaknesses": "In my view, the paper suffers from three main weaknesses:\n\n- The writing in the initial sections of the paper (even the title!) is unnecessarily convoluted and confusing. Initially, I could not understand the main idea of the approach until reading the results sections and the appendix.\n- The experiments are performed on very constrained, synthetic datasets. The train-test split of the audio experiment, especially, seems very simple (the test set contains the same sequences as the training set, just very slightly detuned). For example, based on these experiments, I do not believe the proposed model would work for pitch estimation in-the-wild (whereas SPICE does, to some extent).\n- The results for the proposed approach show a slight or moderate improvement compared to the K=0 baseline. However, most of the observed effect seems to come from the sequence prediction and reconstruction objectives, and not from the proposed physical symmetry (see, e.g., Figure 3, K=0 vs K=4). This is somewhat disappointing. I also believe this fact is not openly discussed in the text.\n\nSome more comments regarding writing:\n- From the title, it is not clear that the approach is applied to the music or image domain. Neither title nor abstract clarify that the proposed approach relies on time-series data. \n- The abstract, intro and conclusion suggest that using (a) using physical symmetry for learning and (b) representation augmentation are two different contributions of the paper, when in fact, they refer to the same thing (i.e., the proposed idea).\n- The term ''prior model'' is used in the abstract, but is not properly explained. In the paragraph starting on line 37, this finally gets explained, but the exposition is again quite convoluted. This would be much easier to understand if the applications used for the experiments (pitch sequences/moving ball) were introduced right at the start.\n- The same goes for ''global invariant style code'', the meaning of which is completely unclear until one reads the appendix. Again, the discussion should be much more concrete and clearly mention what style and content mean in the different application scenarios.\n- Section 2 (Intuition) is actively unhelpful. Instead of a clean explanation (using example applications!), we are given pompous and unscientific prose such as ''we regard symmetry in physical law as a general inductive bias of the human mind'' (line 74).\n- Line 87 introduces the term ''linear pitch factor'', without clarifying the difference between that and simply the term ''pitch''. I suppose the term pitch is not used since the learned latent variables do not encode pitch on a natural (e.g. MIDI) scale, but this only becomes clear to the reader much later in the manuscript.\n- In lines 110f, the reader suddenly learns that the model is a variational autoencoder! This should have been made plain in Section 3.1, or even earlier.\n\nOverall, this is a borderline paper for me, with a tendency towards reject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the detailed feedback and for raising those important questions. In the below chain of comments, we respond in a breakdown format. \n\n> **How important is L_rec, relative to L_prior? That is, can we turn off L_rec and still get similar results?**  \n\n**Reply 1**:  \nNo, we can’t. L_rec is critical in preventing representation collapse. \n\nYour intuition to ablate L_rec is astute in a sense that L_prior is the conceptually central training objective — it literally evaluates the *latent dynamics*. However, without L_rec, the latent representation will collapse, where all possible x is encoded to the same z. In the case of collapse, the prior model will trivially learn the identity function to achieve L_prior=0. Additionally, collapse is guaranteed almost everywhere in the random initialization space since L_prior is a contra**c**tive loss that pulls representations inwards. L_rec forces z to contain at least the information required to reconstruct x, preventing collapse. \n\n> **In Equation 2 (and other equations), how do you use the binary cross entropy loss to compare spectrograms (or pictures)? This is very unusual, since spectrograms do not have a value range [0,1]. Is this applied in an element-wise fashion?**  \n\n**Reply 2**:  \nWe apply it in a pixel-wise fashion. A binary cross entropy (BCE) loss can be used because we normalise the energy value of spectrograms to the range [0, 1] in advance (paper line 144). In fact, we have done experiments comparing using BCE vs. using L_2 loss, and yielded no difference in performance. \n\n> **Line 122: ''In other words, we add or subtract the latent codes by a random scalar.'' -- Unclear. Do you mean you add a scalar *to* the latent code?**  \n\n**Reply 3**:  \nYes, we mean “we add a random scalar to the latent code”. We will update the paper accordingly. \n\n> **Line 137f: Is the choice of sequence (a scale upward and downward) relevant? Could you also choose a stationary sequence? Or a random sequence (transposed to different keys)?**  \n\n**Reply 4**:  \nWe should be able to use random sequences transposed to different keys to train. In contrast, stationary sequences (if you mean all notes in a sequence share the same pitch) will not work Since the RNN would collapse. \n\n> **The results for the proposed approach show a slight or moderate improvement compared to the K=0 baseline. …(see, e.g., Figure 3, K=0 vs K=4).** \n\n**Reply 5**:  \nIt may be a bit misleading if we only focus on Fig 3 in order to understand the effectiveness of SPS. If we take a comprehensive view of the experimental results, i.e. considering Fig 3 alongside Fig 11, it becomes evident that the introduction of timbre factors makes the improvement (of the linearity of pitch) from physical symmetry much more pronounced. As the task difficulty increases, the role of symmetry becomes increasingly significant. \n\n> **From the title, it is not clear that the approach is applied to the music or image domain. Neither title nor abstract clarify that the proposed approach relies on time-series data.**  \n\n**Reply 6**:  \nWe mention physical symmetry in the title and video and audio in the abstract, which implies what we deal with is time-series data. For the sake of clarity for readers, we will indicate time-series  in the revised abstract.\n\n> **The abstract, intro and conclusion suggest that using (a) using physical symmetry for learning and (b) representation augmentation are two different contributions of the paper, when in fact, they refer to the same thing (i.e., the proposed idea).**  \n\n**Reply 7**:  \nWe do not think they are the same things though they are strongly related in this work. Using physical symmetry is a novel methodology to unsupervisedly learn interpretable representation from time-series data, while representation augmentation is just one way to implement it. Of course there might be other pure mathematical ways (as opposed to using imaginary samples) to enforce physical symmetry constraints but we have not found them yet. In addition, representation augmentation can also help methods other than physical symmetry that regularise the latent space. Therefore, we regard them as two different contributions.\n\n> **I think the limited usefulness of the VAE compared to the AE stems from the fact that the datasets used are so simple. There are simply no sources of variability that would need to be modeled by the Gaussian prior.**  \n\n**Reply 8**:  \nIn general, VAE yields better representations than AE even when the data is simple and the Gaussian prior works very well even on non-Gaussian data. Our work indicates that physical symmetry could be an inductive bias which could 1) substitute the gaussian prior for some time-series tasks and 2) at the same time be more interpretable. \n\n> **Section 2 (Intuition) is actively unhelpful. Instead of a clean explanation (using example applications!).**  \n\n**Reply 9**:  \nThe main contribution of the paper is to draw inspiration from modern physics to address a key challenge in machine learning: how to learn interpretable low-dimensional factors from unlabeled data without relying on domain-specific knowledge. Therefore, the writing in the beginning focuses a lot on high-level abstraction and our vision & intuition, which aims to help readers to analogously understand the big picture and why physical symmetry is a general inductive bias. \n\nAbove all, an outstanding paper is not merely a tech report, but also a story with high-level and inspiring ideas. We hope the significance of this text could be reconsidered. \n"}]

Author rebuttal:

Meta-review: Dear Authors,

After extensive deliberation and a comprehensive review of the feedback provided by our esteemed reviewers, we have reached a decision concerning the submission. The paper brings forth a strong and captivating idea that warrants attention in the academic community. Nonetheless, it is evident that certain concerns have remained unaddressed, notably the paper's lack of clarity on the capability of the SPS to learn interpretable representations, especially when incorrect symmetry assumptions are present. 

The scalability of the approach to diverse tasks has been a recurrent concern, and it would have been beneficial to delve deeper into mechanisms to circumvent overfitting. On a constructive note, the discussion surrounding the KL divergence term of the VAE loss, and the potential to encode a symmetry structure into the latent sequence, suggests exciting avenues for future work. The concerns raised about the intertwining of domain knowledge and physical symmetry design, especially in the context of the experiments, demand more nuanced articulation, perhaps by delineating the concept of 'physical symmetry' from its implementation. Lastly, while the use of physical symmetry as a causal inducer remains a point of contention, we believe that further exploration in this direction might provide clarity. The additional feedback on emphasizing the application examples and the autoencoder structure is well-taken. 

Overall, given the potential and significance of the core idea, and with the expectation that the authors will address the highlighted concerns, we are inclined to accept the paper. We believe that the necessary revisions will elevate the paper's value and coherence.

Best reagards,

Area Chair
Decision: Accept (poster)