Title: Learning Interpretable Low-dimensional Representation via Physical Symmetry
Venue: NeurIPS 2023
Accepted: yes
Keywords: Physics Symmetry

Time series data

Self-supervised Learning

Representation Augmentation
Abstract: We have recently seen great progress in learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord and texture. However, most methods rely heavily on music domain knowledge. It remains an open question what general computational principles *give rise to* interpretable representations, especially low-dim factors that agree with human perception. In this study, we take inspiration from modern physics and use *physical symmetry* as a self-consistency constraint for the latent space. Specifically, it requires the prior model that characterises the dynamics of the latent states to be *equivariant* with respect to certain group transformations. We show that physical symmetry leads the model to learn a *linear* pitch factor from unlabelled monophonic music audio in a self-supervised fashion. In addition, the same methodology can be applied to computer vision, learning a 3D Cartesian space from videos of a simple moving object without labels. Furthermore, physical symmetry naturally leads to *counterfactual representation augmentation*, a new technique which improves sample efficiency.

Learning Interpretable Low-dimensional Representation via Physical Symmetry

Abstract

Wehave recently seen great progress in learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord and texture. However, most methods rely heavily on music domain knowledge. It remains an open question what general computational principles give rise to interpretable representations, especially low-dim factors that agree with human perception. In this study, we take inspiration from modern physics and use physical symmetry as a self-consistency constraint for the latent space of time-series data. Specifically, it requires the prior model that characterises the dynamics of the latent states to be equivariant with respect to certain group transformations. We show that physical symmetry leads the model to learn a linear pitch factor from unlabelled monophonic music audio in a self-supervised fashion. In addition, the same methodology can be applied to computer vision, learning a 3D Cartesian space from videos of a simple moving object without labels. Furthermore, physical symmetry naturally leads to counterfactual representation augmentation , a new technique which improves sample efficiency.

1 Introduction

Interpretable representation-learning models have achieved great progress for various types of timeseries data. Taking the music domain as an example, tailored models [Ji et al. , 2020] have been developed to learn pitch, timbre, melody contour, chord progression, texture, etc. from music audio. These human-interpretable representations have greatly improved the performance of generative algorithms in various music creation tasks, including inpainting [Wei et al. , 2022], harmonization [Yi et al. , 2022], (re-)arrangement, and performance rendering [Jeong et al. , 2019].

However, most representation learning models still rely heavily on domain-specific knowledge. For example, to use pitch scales or instrument labels for learning pitch and timbre representations [Luo et al. , 2020, 2019; Engel et al. , 2020; Lin et al. , 2021; Esling et al. , 2018] and to use chords and rhythm labels for learning higher-level representations [Akama, 2019; Yang et al. , 2019; Wang et al. , 2020; Wei and Xia, 2021]. Such an approach is presumably very different from human learning; even without formal music training, we see that many people can learn pitch , a fundamental music concept, simply from the experience of listening to music. Hence, it remains an open question how to learn interpretable pitch factor using inductive biases that are more general. In other words, what general computational principle gives rise to the concept of pitch.

We see a similar issue in other domains. For instance, various computer-vision models [McCarthy and Ahmed, 2020; Trevithick and Yang, 2021; Mescheder et al. , 2019; Riegler et al. , 2017] can learn

3D representations of human faces or a particular scene by using domain knowledge (e.g., labelling of meshes and voxels, 3D convolution, etc.) But when these domain setups are absent, it remains a non-trivial task to learn the 3D location of a simple moving object in a self-supervised fashion.

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

-1

GLYPH(cmap:d835)

-1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Inspired by modern physics, we explore to use physical symmetry (i.e., symmetry of physical laws) as a weak self-consistency constraint for the learned latent z space of time-series data x . As indicated in Figure 1, this general inductive bias requires the learned prior model R , which is the induced physical law describing the temporal flow of the latent states, to be equivariant to a certain transformation S (e.g., translation or rotation). Formally, z t +1 = R ( z t ) if and only if z S t +1 = R ( z S t ) , where z S = S ( z ) . In other words, R and S are commutable for z , i.e., R ( S ( z )) = S ( R ( z )) . Note that our equivariant assumption applies only to the latent z space. This is fundamentally different from most existing symmetry-informed models [Bronstein et al. , 2021], in which the equivariant property also imposes assumptions on the raw data space.

Specifically, we design s elf-supervised learning with p hysical s ymmetry ( SPS ) 1 , a method that adopts an encoder-decoder framework and applies physical symmetry to the prior model. We show that SPS learns a linear pitch factor (that agree with human music perception) from monophonic music audio without any domain-specific knowledge about pitch scales, f0, or harmonic series. The same methodology can be applied to the computer vision domain, learning 3D Cartesian space from monocular videos of a bouncing ball shot from a fixed perspective. In particular, we see four desired properties of SPS as a self-supervised algorithm for interpretability:

Conciseness : SPS does not require contrastive samples, bach normalization, or a large batch size. We can even drop the Gaussian prior regularization, which is usually required to learn a meaningful latent space. (See Section 3.2 - Section 4.)

Sample efficiency : SPS learns low-dimensional representations of a dynamic system in a very sample-efficient way. (See Section 4 - Section 5.2.)

Robustness : Even with an incorrect symmetry assumption, SPS can still learn more interpretable representations than baselines. (See Section 5.3.)

Extendability : SPS can be easily combined with other learning techniques. For example, if we further assume an extra global invariant style code, the model becomes a disentangled sequential autoencoder, capable of learning content-style disentanglement from temporal signals. (See appendix.)

2 Intuition

The idea of using physical symmetry for representation learning comes from modern physics. In classical physics, scientists usually first induce physical laws from observations and then discover symmetry properties of the law. (E.g., Newton's law of gravitation, which was induced from planetary orbits, is symmetric with respect to Galilean transformations.) In contrast, in modern physics, scientists often start from a symmetry assumption, based on which they derive the corresponding law and predict the properties (representations) of fundamental particles. (E.g., general relativity was developed based on a firm assumption of symmetry with respect to Lorentz transformations).

Analogously, we use physical symmetry as an inductive bias of our representation learning model, which helps us learn a regularised prior and an interpretable low-dim latent space. If it is a belief of many physicists that symmetry in physical law is a major design principle of nature, we regard symmetry in physical law as a general inductive bias of perception. In other words, if physical symmetry leads an AI agent to learn human-aligned concepts in a self-supervised fashion, we believe that it could also provide insights into the ways that human minds perceive the world.

The introduction of physical symmetry naturally leads to counterfactual representation augmentation , a novel learning technique which helps improve sample efficiency. Representation augmentation means to 'imagine' extra pairs of z S t and z S t +1 as training samples for the prior model R . Through the lens of causality, this augmentation can be seen as a counterfactual inductive bias of the prediction on the representation level what if the prior model makes predictions based on transformed latent codes? As indicated in Figure 1, it requires the prediction of the z sequence to be equivariant to certain group transformations, S . This regularisation also constrains the encoder and decoder indirectly through the prior model since the network is trained in an end-to-end fashion.

3 Methodology

With physical symmetry, we aim to learn an interpretable low-dimensional representation z i of each high-dimensional sample x i from time-series x 1: T . We focus on two problems in this paper: 1) to learn a 1D linear pitch factor of music notes from music audio, where each x i is a spectrogram of a note, and 2) to learn 3D Cartesian location factors of a simple moving object (a bouncing ball) from its trajectory shot by a fixed, single camera, where each x i is an image.

3.1 Model

Figure 2 shows the model design of SPS. During the training process, the temporal data input x 1: T is first fed into the encoder E to obtain the corresponding representation z 1: T . Then it is fed into three branches. In the first branch (the green line), z 1: T is decoded directly by the decoder D to reconstruct x ′ 1: T . In the second branch (the orange line), z 1: T is passed through the prior model R to predict its next timestep, ˆ z 2: T +1 , which is then decoded to reconstruct ˆ x 2: T +1 . In the third branch (the blue line), we transform z 1: T with S , pass it through R , and transform it back using the inverse transformation S -1 to predict another version of the next timestep ˜ z 2: T +1 , and finally decode it to ˜ x 2: T +1 . We get three outputs from the model: x ′ 1: T , ˆ x 2: T +1 , and ˜ x 2: T +1 .

The underlying idea of physical symmetry is that the dynamics of latent factor and its transformed version follow the same physical law characterised by R . Therefore, ˜ z and ˆ z should be close to each other and so are ˜ x and ˆ x , assuming S is a proper transformation. This self-consistency constraint helps the network learn a more regularised latent space.

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

GLYPH(cmap:d835)

-1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

or

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

′

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

3.2 Training objective

The total loss contains four terms: reconstruction loss L rec, prior prediction loss L prior, symmetrybased loss L sym, and KL divergence loss L KLD. Formally,

where λ 1 , λ 2 and λ 3 are weighting parameters. By referring to the notations in section 3.1,

L KLD is the Kulback-Leibler divergence loss between the prior distribution of z i and a standard Gaussian. Lastly, we build two versions of SPS: SPSVAE and SPSAE, with the latter replacing the VAE with an AE (and trivially doesn't have L KLD).

3.3 Symmetry-based counterfactual representation augmentation

During training, S is the counterfactual representation augmentation since it creates extra imaginary sequences of z (i.e., imaginary experience) to help train the prior. In practice, for each batch we apply K different transformations S 1: K to z and yield K imaginary sequences. Thus, the two terms of symmetry-based loss can be specified as:

where the lower case k denotes the index of a specific transformation and we refer to K as the augmentation factor . Likewise, the last term of reconstruction loss can be specified as:

Each S applied to each sequence z 1: T belongs to a certain group, and different groups are used for different problems. For the music problem, we assume z i be to 1D and use random S ∈ G ∼ = ( R , +) . In other words, we add a random scalar to the latent codes. As for the video problem, we assume z i be to 3D and use random S ∈ G ∼ = ( R 2 , +) × SO(2) . In other words, random rotation and translation are applied on two dimensions of z i .

4 Results

We test SPS under two modalities of temporal signals: music (section 4.1) and video (section 4.2). Each model is executed with 10 random initialisations, and evaluated on the test set. The highlight of this section is that SPS effectively learns interpretable low-dimensional factors that align with human perceptions. Also, by utilizing small training sets, we show the high sampling efficiency of our model. In the appendix, we further show that SPS also maintains accuracy in both reconstruction and prediction tasks (section A.2). Additionally, we present supplementary results trained on more complicated datasets and a more advanced configuration of our model, called SPS+, which enables content-style disentanglement in addition to interpretable learning. The findings from these more complex scenarios align closely with those observed in the simpler cases presented in this section.

4.1 Learning linear pitch factor from music audio

4.1.1 Dataset and training setups

We synthesise a training dataset of 27 audio clips, each containing 15 notes in major scales with the first 8 notes ascending and the last 8 notes descending. We vary the starting pitch by integer numbers of MIDI pitch such that every MIDI pitch in the range A#4 to C7 is present in the training set. Only the accordion is used to generate the clips. For each clip in the training set, we uniformly randomly shift its pitch upwards by a decimal between 0 and 1, in this way generate the test set for evaluation.

We convert each audio clip into a sequence of image segments for processing. First, we run STFT (with sample rate = 16000 /s , window length = 1024 , hop length = 512 , with no padding and no logarithmic frequency scale) over each audio clip to obtain a power spectrogram. After normalising the energy to the range [0 , 1] , we slice the power spectrogram into fifteen image segments, each containing one note. The CNN encoder, in each timestep, takes one segment as input. For the latent space, we assume z i ∈ R and sample counterfactual representation augmentation S ∼ U ([ -1 , 1]) where S ∈ G ∼ = ( R , +) . Note this assumption does not indicate any domain-specific inductive bias of music, such as the logarithmic relationship between pitch and frequency or the relationship between F0 and harmonics.

4.1.2 Results on interpretable pitch space

C6

Pitch

-

5

0

z

Embedding

SPS

VAE

K

=4 (Ours)

C6

Pitch

0

5

SPS

AE

K

=4 (Ours)

C6

Pitch

0

1

2

SPS

VAE

K

=0 (Ablation)

C6

Pitch

0

1

2

3

SPS

AE

K

=0 (Ablation)

C6

Pitch

0

2

β

-VAE

C6

Pitch

0

.

6

0

.

8

SPICE

-

10

0

z

C6

Synthesis

0

10

z

C6

0

.

0

2

.

5

z

C6

0

.

0

2

.

5

z

C6

0

.

0

2

.

5

z

C6

Figure 3 demonstrates that the 1D pitch factor learned by our model exhibits a linear relationship with the conventional numerical ordering used to represent pitch by humans (e.g. MIDI pitch numbers). The plot shows the mappings of two tasks and six models. In the embedding task (the first row), the x -axis is the true pitch and the y -axis is embedded z . In the synthesis task (the second row), the x -axis is z and the y -axis is the detected pitch (by YIN algorithm, a standard pitch-estimation method by [De Cheveigné and Kawahara, 2002]) of decoded (synthesised) notes. The first two models are SPS based on VAE and AE, respectively, trained with counterfactual representation augmentation factor K = 4 . The third and fourth models are trained without constraints of physical symmetry ( K = 0 ), serving as our ablations. The fifth one is a vanilla β -VAE, trained only to reconstruct, not to predict. The last one is SPICE [Gfeller et al. , 2020], a SOTA unsupervised pitch estimator with strong domain knowledge on how pitch linearity is reflected in log-frequency spectrograms . As the figure shows, 1) without explicit knowledge of pitch, our model learns a more interpretable pitch factor than β -VAE, and the result is comparable to SPICE, and 2) without the Gaussian prior assumption of latent variable distribution, our model SPSAE also learns a continuous representation space.

Table 1 shows a more quantitative analysis using R 2 as the metric to evaluate the linearity of the pitch against z mapping from the encoder and the decoder. All models except SPICE are trained with 10 random initializations.

4.2 Learning object 3D coordinates from videos of a moving object

4.2.1 Dataset and training setups

We run physical simulations of a bouncing ball in a 3D space. The ball is randomly thrown and affected by gravity and the bouncing force (elastic force). A fixed camera records a 20-frame video of each 4-second simulation to obtain one trajectory (see Figure 4). The ball's size, gravity, and proportion of energy loss per bounce are constant across all trajectories. For each trajectory, the ball's initial location and initial velocity are randomly sampled. We utilize 512 trajectories for training, and an additional 512 trajectories for evaluation.

For the latent space, we set the dimension of the latent space to 3, but only constrain 2 of them by augmenting the representations with S ∈ G ∼ = ( R 2 , +) × SO(2) . Those two dimensions are intended to span the horizontal plane. The third one, which is the unaugmented latent dimension, is intended to encode the vertical height.

4.2.2 Results on interpretable 3D representation

Figure 5 visually evaluates the interpretability of the learned location factors by traversing the z space, one dimension at a time, and using the learned decoder to synthesise images. If the learned factors are linear w.r.t. the 3D Cartesian coordinates, the synthesised ball should display linear motions as we change z linearly. In brief, SPS learns an a more interpretable and linear z space. Here, subplot (a) depicts the results of SPSVAE with K =4. We see that the un-augmented dimension, z 2 , controls the height of the ball, while z 1 and z 3 move the ball along the horizontal (ground) plane. Each axis is much more linear than in (b) and (c). Subplot (b) evaluates SPSVAE with counterfactual representation augmentation K =0, essentially turning off SPS. As z i varies, the ball seems to travel along curves in the 3D space, showing the ablation learns some continuity w.r.t. the 3D space, but is obviously far from linear. In (c), the β -VAE fails to give consistent meaning to any axis.

Table 2 further shows quantitative evaluations on the linearity of the learned location factor, in which we see that SPS outperforms other models by a large margin. To measure linearity, we fit a linear regression from z to the true 3D location over the test set and then compute the Mean Square Errors (MSE). Therefore, a smaller MSE indicates a better fit. To give an intuitive example, the MSEs of (a), (b) and (c) in Figure 5 are 0.09, 0.58 and 0.62 respectively. Here, we also include the results of SPSAE. Very similar to the music experiment in session 4.1, we again see that even without the Gaussian prior assumption, our model SPSAE learns an interpretable latent space comparable to SPSVAE.

z

1

z

2

2

σ

z

3

0

+2

σ

(a) SPS

VAE

,

K

=4 (Ours)

z

1

z

2

2

σ

z

3

0

+2

σ

(b) SPS

VAE

,

K

=0 (Ablation)

z

1

z

2

2

σ

z

3

0

+2

σ

(c)

β

-VAE (Baseline)

̸

-

-

-

5 Analysis

To better understand the effects of counterfactual representation augmentation (first introduced in section 3.3), we ran extra experiments with different S and K . We choose the vision problem since a 3D latent space manifests a more obvious difference when physical symmetry is applied. In section 5.1, we show that a larger augmentation factor K leads to higher sample efficiency. In section 5.2, we visualise the change of learned latent space against training epoch according to different values of K . In section 5.3, we show that some deliberately incorrect group assumptions S can also achieve good results.

5.1 Counterfactual representation augmentation improves sample efficiency

Figure 6 shows that a larger factor of counterfactual representation augmentation leads to a lower linear projection loss (the measurement defined in section 4.2.2) of the learned 3D representation. Here, K is the augmentation factor, and K = 0 means the model is trained without physical symmetry. The comparative study is conducted on 4 training set sizes (256, 512, 1024, and 2048), in which each box plot shows the results of 10 experiments trained with a fixed K and random initialisation. We see that a larger K leads to better results and compensates for the lack of training data. E.g., the loss trained on 256 samples with K = 4 is comparable to the loss trained on 1024 samples with K = 0 , and the loss trained on 512 samples with K = 4 is even lower than the loss trained on 2048 samples with K = 0 . Furthermore, when K = 0 , increasing the number of training samples beyond a certain point does not further shrink the error, but increasing K still helps.

5.2 Counterfactual representation augmentation improves interpretability

Figure 7 visualises the latent space during different stages of model training, and we see that a larger K leads to a better enforcement of interpretability. The horizontal axis shows the training epoch. Three experiments with different K values ( × 0 , × 4 , × 16 ) are stacked vertically. Each experiment is

0

4

16

K

0

.

0

0

.

2

0

.

4

0

.

6

Linear proj.

loss

↓

256 Samples

0

4

16

K

512 Samples

0

4

16

K

1024 Samples

0

4

16

K

2048 Samples

-

2

2

-

2

2

-

2

2

-

2

2

-

2

2

-

2

2

-

2

2

331

Epoch

-

2

2

669

-

2

2

1000

-

2

2

1331

-

2

2

1669

-

2

2

2000

-

2

2

2331

-

2

2

2669

-

2

2

3000

-

2

2

3331

-

2

2

3669

-

2

2

4000

-

2

2

4331

-

2

2

4669

-

2

2

5000

SPS

VAE

,

K

SPS

VAE

,

K

=4

=0

SPS

VAE

,

K

=16

trained twice with random initialisation. Each subplot shows the orthogonal projection of the z space onto the plane spanned by z 1 and z 3 , therefore hiding most of the y -axis (i.e. ball height) wherever a linear disentanglement is fulfilled. During training, the role of physical symmetry is to 'straighten' the encoded grid and a larger K yields a stronger effect.

5.3 Counterfactual representation augmentation with deliberately incorrect group assumptions

Additionally, we test SPS with deliberately incorrect group assumptions. The motivation is as follows. In real applications, researchers may incorrectly specify the symmetry constraint when the data are complex or the symmetry is not known a priori . SPS is more useful if it works with various groups assumptions close to the truth. In our analysis, we are surprised to find that SPS still learns interpretable representations under alternate group assumptions via perturbing the correct one.

Figure 8 shows our results with the vision task (on the bouncing ball dataset). The x tick labels show the augmentation method. Its syntax follows section 3.3, e.g., ' ( R 1 , +) × SO(2) ' denotes augmenting representations by 1D translations and 2D rotations. The y axis of the plot is still linear projection loss (as discussed in section A.5.3) that evaluates the interpretability of the learned representation. As is shown by the boxplot, five out of five perturbed group assumptions yield better results than the 'w/o Symmetry' baseline. Particularly, ( R 3 , +) × SO(2) and ( R 2 , +) × SO(3) learn significantly more linear representations, showing that some symmetry assumptions are 'less incorrect' than others, and that SPS can achieve good results under a multitude of group assumptions.

w/o

Symmetry

(

R

2

,

+)

SO(2)

(

R

1

,

+)

×

SO(2)

(

R

3

,

+)

×

SO(2)

(

R

2

,

+)

×

SO(3)

(

R

2

,

+)

×

SO(2)

(Ours)

0

.

0

0

.

2

0

.

4

0

.

6

Linear proj.

loss

6 Related work

The idea of using a predictive model for better self-supervised learning has been well established [Oord et al. , 2018; Chung et al. , 2015; LeCun, 2022]. In terms of model architecture, our model is very similar to VRNN [Chung et al. , 2015]. In addition, our model can be seen as a variation of joint-embedding predictive architecture (JEPA) in [LeCun, 2022] if we eliminate the reconstruction losses on the observation. In fact, we see the network topology of a model as the 'hardware' and see the learning strategy (e.g., contrastive method, regularised method, or a mixed one) as the 'software'. The main contribution of this study lies in the learning strategy - to use physical symmetry to limit the complexity of the prior model, and to use counterfactual representation augmentation to increase sample efficiency.

The existing notation of 'symmetry' as in [Higgins et al. , 2018; Bronstein et al. , 2021] is very different from physical symmetry as an inductive for representation learning . Most current symmetrybased methods care about the relation between observation x and latent z [Sanghi, 2020; Quessard et al. , 2020; Dupont et al. , 2020; Huang et al. , 2021]. E.g., when a certain transformation is applied to x , z should simply keep invariant or follow a same/similar transformation. Such an assumption inevitably requires some knowledge in the domain of x . In contrast, physical symmetry focuses solely on the dynamics of z , and therefore we only have to make assumptions about the underlying group transformation in the latent space. We see two most relevant works in the field of reinforcement learning [Mondal et al. , 2022; Dupont et al. , 2020], which apply an equivariant assumption similar to the physical symmetry used in this paper. The major differences are twofold. First, to disentangle the basic factors, our method requires no interactions with the environment. Second, our method is much more concise; it needs no other tailored components or other inductive biases such as symmetric embeddings network and contrastive loss used in [Dupont et al. , 2020] or MDP homomorphism applied in [Mondal et al. , 2022].

7 Limitation

We have identified several limitations in the generality and soundness of SPS. Firstly, when the underlying concept following physical symmetry only contains partial information of the time series and cannot fully reconstruct the inputs, SPS may not function properly. We hypothesize that this issue is connected to content-style disentanglement, and present some preliminary results in appendix A.4 and A.5. Secondly, the current model lacks the ability to distill concepts from multibody systems. For example, it is unable to learn the concept of pitch from polyphonic music or understand 3D space from videos featuring multiple moving objects. Lastly, it is essential to develop a formalized theory for quantifying the impact of counterfactual representation augmentation in future work. This would involve measuring the degree of freedom in the latent space with and without physical symmetry, and explaining why incorrect symmetry assumptions can still result in a correct and interpretable latent space.

8 Conclusion

In this paper, we use physical symmetry as a novel inductive bias to learn interpretable and lowdimensional representations from time-series data. Experiments show that physical symmetry effectively distills an interpretable linear pitch concept, which agrees with human music perception, from music audios without any labels. With the same method, we can learn the concept of 3D Cartesian space from monocular videos of bouncing ball shot from a fixed perspective. In addition, a robust training technique, counterfactual representation augmentation, is developed to enforce physical symmetry during training. Analysis shows that counterfactual representation augmentation leads to higher sample efficiency and better latent-space interpretability, and it stays effective even when the symmetry assumption is incorrect. Last but not least, we see that with physical symmetry, our sequential representation learning model can drop the the Gaussian prior regulation on the latent space. Such a result empirically indicates that physical symmetry, as a causal (counterfactual) inductive bias, might be more essential compared to the Gaussian prior as a purely statistical regularization.

Acknowledgments

We'd like to thank to Dr. Zhao Yang and Dr. Maigo Wang for inspiring discussion on physical symmetry. Also, we'd like to like to extend our sincere thanks to Junyan Jiang and Yixiao Zhang for their useful and practical suggestions.

References

Taketo Akama. Controlling symbolic music generation based on concept learning from domain knowledge. In Arthur Flexer, Geoffroy Peeters, Julián Urbano, and Anja Volk, editors, Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019 , pages 816-823, 2019.

Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems , 34:10105-10118, 2021.

Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems , 28, 2015.

Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 , 2021.

Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation , pages 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics.

Collins Chris. GeneralUser GS, 2017.

Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Alain De Cheveigné and Hideki Kawahara. Yin, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America , 111(4):1917-1930, 2002.

Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning , pages 2761-2770. PMLR, 2020.

Jesse Engel, Rigel Swavely, Lamtharn Hanoi Hantrakul, Adam Roberts, and Curtis Hawthorne. Selfsupervised pitch detection by inverse audio synthesis. In ICML 2020 Workshop on Self-supervision in Audio and Speech , 2020.

Philippe Esling, Axel Chemla-Romeu-Santos, and Adrien Bitton. Bridging audio analysis, perception and synthesis with perceptually-regularized variational timbre spaces. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018 , pages 175-181, 2018.

Eric Foxley. Nottingham database, 2011.

Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Velimirovi´ c. Spice: Self-supervised pitch estimation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 28:1118-1128, 2020.

Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230 , 2018.

Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in Neural Information Processing Systems , 30, 2017.

Siyuan Huang, Yichen Xie, Song-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised representation learning for 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6535-6545, 2021.

Dasaem Jeong, Taegyun Kwon, Yoojin Kim, and Juhan Nam. Graph neural network for music score data and modeling expressive piano performance. In International Conference on Machine Learning , pages 3060-3070. PMLR, 2019.

Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey on deep music generation: Multi-level representations, algorithms, evaluations, and future directions. arXiv preprint arXiv:2011.06801 , 2020.

David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations , 2021.

Yann LeCun. A path towards autonomous machine intelligence. preprint posted on openreview , 2022.

Liwei Lin, Gus Xia, Qiuqiang Kong, and Junyan Jiang. A unified model for zero-shot music source separation, transcription and synthesis. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy, editors, Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021 , pages 381-388, 2021.

Yin-Jyun Luo, Kat Agres, and Dorien Herremans. Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders. Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019 , 2019.

Yin-Jyun Luo, Kin Wai Cheuk, Tomoyasu Nakano, Masataka Goto, and Dorien Herremans. Unsupervised disentanglement of pitch and timbre for isolated musical instrument sounds. In Proceedings of the 21st International Society for Music Information Retrieval Conference, ISMIR 2020 , pages 700-707, 2020.

Yin-Jyun Luo, Sebastian Ewert, and Simon Dixon. Towards robust unsupervised disentanglement of sequential data - a case study using music audio. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 , pages 3299-3305. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.

Ollie McCarthy and Zohaib Ahmed. Hooligan: Robust, high quality neural vocoding. arXiv preprint arXiv:2008.02493 , 2020.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4460-4470, 2019.

Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh. Eqr: Equivariant representations for data-efficient reinforcement learning. In International Conference on Machine Learning , pages 15908-15926. PMLR, 2022.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems , 33:19727-19737, 2020.

Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3577-3586, 2017.

Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maximization and contrastive learning. In European Conference on Computer Vision , pages 626-642. Springer, 2020.

Alex Trevithick and Bo Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15182-15192, 2021.

Matthew J Vowels, Necati Cihan Camgoz, and Richard Bowden. Vdsm: Unsupervised video disentanglement with state-space modeling and deep mixtures of experts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8176-8186, 2021.

Ziyu Wang, Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Julie Cumming, Jin Ha Lee, Brian McFee, Markus Schedl, Johanna Devaney, Cory McKay, Eva Zangerle, and Timothy de Reuse, editors, Proceedings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020 , pages 662-669, 2020.

Shiqi Wei and Gus Xia. Learning long-term music representations via hierarchical contextual constraints. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy, editors, Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021 , pages 738-745, 2021.

Shiqi Wei, Gus Xia, Yixiao Zhang, Liwei Lin, and Weiguo Gao. Music phrase inpainting using longterm representation and contrastive loss. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 186-190, 2022.

Frank Wen. The fluid release 3 general-MIDI soundfont, 2013.

Ruihan Yang, Dingsu Wang, Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. In Arthur Flexer, Geoffroy Peeters, Julián Urbano, and Anja Volk, editors, Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019 , pages 596-603, 2019.

Li Yi, Haochen Hu, Jingwei Zhao, and Gus Xia. Accomontage2: A complete harmonization and accompaniment arrangement system. In Ismir 2022 Hybrid Conference , 2022.

Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning , pages 5670-5679. PMLR, 2018.

Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6538-6547, 2020.

A Appendix

The appendix is structured into three main parts.

The first part (section A.1, A.2) provides additional details about SPS. Section A.1 focuses on implementation-related aspects, while section A.2 presents experimental results concerning the reconstruction and prediction loss.

The second part (section A.3 to A.5) introduces an extended version of SPS called SPS+. Section A.3 describes the capabilities of SPS+ in achieving content-style disentanglement along with interpretable learning. The related experiments are presented in section A.4 and section A.5.

The third part (section A.6) presents two additional complex experiments conducted separately using SPS+ and SPS, respectively.

A.1 SPS implementation details

A.1.1 Architecture details

Our models for both tasks share the following architecture. The encoder first uses a 2D-CNN with ReLU activation to shrink the input down to an 8 × 8 middle layer, and then a linear layer to obtain z . If the encoder is in a V AE (instead of an AE), two linear layers characterises the posterior, one for the mean and the other for the log-variance. The prior model is a vanilla RNN of one layer with 256 hidden units and one linear layer projection head. The decoder consists of a small fully-connected network followed by 2D transposed convolution layers mirroring the CNN in the encoder. Its output is then passed through a sigmoid function. We use no batch normalisation or dropout layers.

Minor variations exist between the models for the two tasks. In the audio task, we use three convolution layers in the encoder, with three linear and three 2D transposed convolution layers in the decoder. In the vision task, as the data are more complex, we use four convolution layers in the encoder, with four linear and four 2D transposed convolution layers in the decoder.

A.1.2 Training details

For both tasks, we use the Adam optimiser with learning rate = 10 -3 . The training batch size is 32 across all of our experiments. For all V AE-based models, including SPSVAE (ours/ablation) and β -VAE (baseline), we set β (i.e., λ 3 in Equation (1)) to 0 . 01 , with λ 1 = 1 and λ 2 = 2 . All BCE and MSE loss functions are calculated in sum instead of mean. K = 4 for all SPS models except for those discussed in section 5 where we analyse the influence of different K .

The RNN predicts z n +1: T given the first n embeddings z 1: n . We choose n = 3 for the audio task and n = 5 for the vision task. We adopt scheduled sampling [Bengio et al. , 2015] during the training stage, where we gradually reduce the guidance from teacher forcing. After around 50000 batch iterations, the RNN relies solely on the given z 1: T and predicts auto-regressively.

A.2 SPS reconstruction and prior prediction results

We investigate the reconstruction and prediction capacities of our model and show that they are not harmed by adding symmetry constraints. For the music task, we compare our model, our model ablating symmetry constraints, and a β -VAE trained solely for the reconstruction of power spectrogram. Table 3 reports per-pixel BCE of the reconstructed sequences from the original input frames (Self-recon) and from the RNN predictions (Image-pred). We also include L prior , the MSE loss on the RNN-predicted ˆ z as defined in section 3.2. The results show that our models slightly surpasses the ablation and baseline models in all three metrics.

Similarly, Table 4 displays the reconstruction and prediction losses on the test set for the video task. Results show that adding symmetry constraints does not significantly hurt the prediction losses. Frame-wise self-reconstruction is significantly lower for the SPS models, but only by a small margin.

A.3 SPS+

SPS can use physical symmetry to learn interpretable factors that evolve over time. We call those factors content representation. However, many problems can not be represented by content represen-

±

tation alone. For example, the bouncing balls can have different colours and the pitch scales can be generated by different instruments. If the colour of a ball or the timbre of a sound scale are constant within a trajectory, those latent spaces are hard to constrain by physical symmetry. We call such invariant factors style representation. In order to deal with these problems, we combine SPS with a simple content-style disentanglement technique: SPS+, a more general framework of SPS. We use random pooling to constrain the style factors, and use physical symmetry to constrain the content representation in the same way as SPS in Section 3.1

A.3.1 Model

Figure 9 shows the design of SPS+, which belongs to the family of disentangled sequential autoencoders [Bai et al. , 2021; Hsu et al. , 2017; Vowels et al. , 2021; Yingzhen and Mandt, 2018; Zhu et al. , 2020]. During the training process, the temporal data input x 1: T is first fed into the encoder E to obtain the corresponding representation z 1: T . z 1: T is then split into two parts: the style factor z 1: T,s and the content factor z 1: T,c . The style factor z 1: T,s is passed through the random-pooling module P , where one element z τ,s is randomly picked. The content factor z 1: T,c is fed into three branches, then combined with z τ,s to reconstruct. For random pooling in the training stage, one style vector is randomly selected from all time steps (i.e., 15 for the music task and 20 for the vision task) of the sequence to represent z s . In the testing stage, only the first 5 (vision task) or 3 (music task) frames are given, and z s will be selected from them.

A.3.2 Training objective

The following loss functions in SPS+ slightly vary from those in SPS. For SPS+, L prior and L sym work on the content part of latent variables only. Other loss functions are exactly the same as those defined in section 3.2

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

GLYPH(cmap:d835)

-1

split

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

or

GLYPH(cmap:d835)

1:GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1,GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1,GLYPH(cmap:d835)

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1,GLYPH(cmap:d835)

GLYPH(cmap:d835)

1:GLYPH(cmap:d835)

′

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

GLYPH(cmap:d835)

2:GLYPH(cmap:d835)+1

A.4 SPS+ on learning pitch & timber factors from audios of multiple instruments

A.4.1 Dataset and setups

We synthesise a dataset that contains around 2400 audio clips played by multiple instruments . Similar to the dataset in section 4.1.1, each clip contains 15 notes in major scales with the first 8 notes ascending and the last 8 notes descending. Each note has the same volume and duration. The interval between every two notes is equal. We vary the starting pitch such that every MIDI pitch in the range C2 to C7 is present in the dataset. For each note sequence, we synthesise it using 53 different instruments, yielding 2376 audio clips. Specifically, two soundfonts are used to render those audio clips respectively: FluidR3_GM [Wen, 2013] for the train set and GeneralUser GS v1.471 [Chris, 2017] for the test set. The pitch ranges for different instruments vary, so we limit each instrument to its common pitch range (See Table 14).

We assume z c ∈ R and z s ∈ R 2 , and use random S ∈ G ∼ = ( R , +) to augment z c with K =4.

A.4.2 Results on pitch-timbre disentanglement

Weevaluate the content-style disentanglement using factor-wise data augmentation following [Yang et al. , 2019]. Namely, we change (i.e., augment) the instrument (i.e., style) of notes while keeping their pitch the same, and then measure the effects on the encoded z c and z s . We compare the normalised z c and z s , ensuring they have the same dynamic range. Ideally, the change of z s should be much more significant than z c . Here, we compare four approaches: 1) our model (SPS+), 2) our model without splitting for z s (SPS with z ∈ R 3 and S ∈ G ∼ = ( R , +) ) as an ablation, 3) GMVAE [Luo et al. , 2019], a domain-specific pitch-timbre disentanglement model trained with explicit pitch labels , and 4) TS-DSAE [Luo et al. , 2022], a recent unsupervised pitch-timbre disentanglement model based on Disentangled Sequential Autoencoder (DSAE).

Figure 10 presents the changes in normalised z c and z s measured by L2 distance when we change the instrument of an anchor note whose pitch is D3 and synthesised by accordion. Table 5 provides a more quantitative version by aggregating all possible instrument combinations and all different

Clarinet

Flute

Guitar

Piano

Saxophone

Trumpet

Violin

0

2

SPS

VAE

+ (Ours)

Clarinet

Flute

Guitar

Piano

Saxophone

Trumpet

Violin

0

1

SPS

VAE

(Ablation)

Clarinet

Flute

Guitar

Piano

Saxophone

Trumpet

Violin

0

.

0

2

.

5

5

.

0

GMVAE (Baseline)

Clarinet

Flute

Guitar

Piano

Saxophone

Trumpet

Violin

0

5

TS-DSAE (Baseline)

||

∆

z

c

||

2

||

∆

z

s

||

2

pitch pairs. Both results show that SPS+ produces a smaller relative change in z c under timbre augmentation, demonstrating a successful pitch-timbre disentanglement outperforming both the ablation and baseline. Note that for the ablation model, z c varies heavily under timbre augmentation, seemingly containing timbre information. This result indicates that the design of an invariant style factor over the temporal flow is necessary to achieve good disentanglement.

Wefurther quantify the results in the form of augmentation-based queries following [Yang et al. , 2019], regarding the intended split in z as ground truth and the dimensions with the largest variances from factor-wise augmentation after normalisation as predictions. For example, under timbre augmentation under a given pitch for our model, if z 1 and z 3 are the two dimensions of z that produce the largest variances after normalisation, we count one false positive ( z 1 ), one false negative ( z 2 ), and one true positive ( z 3 ). The precision would be 0.67. Table 6 shows the precision scores of the four approaches against their corresponding random selection. The results are in line with our observation in the previous evaluation, with our model more likely to produce the largest changes in dimensions in z c under content augmentation and that in z s under style augmentation.

C2

C4

C6

Pitch

-

5

0

5

z

pitch

Embedding

SPS

VAE

+,

K

=4 (Ours)

C2

C4

C6

Pitch

-

2

0

2

SPS

VAE

+,

K

=0 (Ablation)

C2

C4

C6

Pitch

-

2

.

5

0

.

0

2

.

5

β

-VAE (Baseline)

C2

C4

C6

Pitch

0

.

25

0

.

50

0

.

75

SPICE (Baseline)

-

5

0

5

z

pitch

C2

C4

C6

Synthesis

-

2

.

5

0

.

0

2

.

5

z

pitch

C2

C4

C6

0

.

0

2

.

5

z

pitch

C2

C4

C6

Accordion

Saxophone

Clarinet

Trumpet

Guitar

Violin

Piano

Flute

A.4.3 Results on interpretable pitch space

Figure 11 shows that the pitch factor learned by SPS+ has a linear relation with the true pitch. Here, we use z pitch as the synonym of z c to denote the content factor. The plot shows the mappings of two tasks and four models. In the embedding task (the first row), x -axis is the true pitch and y -axis is embedded z pitch . In the synthesis task (the second row), x -axis is z pitch and y -axis is the detected pitch (by YIN algorithm, a standard pitch-estimation method by [De Cheveigné and Kawahara, 2002]) of decoded (synthesised) notes. The fours models involved are: 1) our model, 2) our model without symmetry ( K =0), 3) a β -VAE trained to encode single-note spectrograms from a single instrument (banjo) to 1D embeddings, and 4) SPICE [Gfeller et al. , 2020], a SOTA unsupervised pitch estimator with strong domain knowledge on how pitch linearity is reflected in log-frequency spectrograms . As the figure shows, without explicit knowledge of pitch, our model learns a more interpretable pitch factor than β -VAE, and the result is comparable to SPICE.

Figure 12 shows a more quantitative analysis, using R 2 as the metric to evaluate the linearity of the pitch against z pitch mapping. Although SPICE produces rather linear mappings in Figure 11, it suffers from octave errors towards extreme pitches, hurting its R 2 performance.

A.4.4 Reconstruction and prior prediction

We investigate the reconstruction and prediction capacities of our model and show that they are not harmed by adding symmetry constraints. We compare our model, our model ablating symmetry constraints, and a β -VAE trained solely for only image reconstruction. Table 7 reports per-pixel BCE of the reconstructed sequences from the original input frames (Self-recon) and from the RNN predictions (Image-pred). We also include L prior , the MSE loss on the RNN-predicted ˆ z as redefined in section A.3.2. The results show that our model surpasses the ablation and baseline models in all three indexes.

0

.

0

0

.

5

1

.

0

R

2

Embedding

SPS

VAE

+,

K

=4 (Ours)

β

-VAE (Baseline)

SPICE (Baseline)

Piano

Accordion

Clarinet

Flute

Guitar

Saxophone

Trumpet

Violin

0

.

0

0

.

5

1

.

0

R

2

Synthesis

A.5 SPS+ on learning space & colour factors from videos of colourful bouncing balls

A.5.1 Dataset and setups

We run physical simulations of a bouncing ball in a 3D space and generate 4096 trajectories, yielding a dataset of videos. Similar to the dataset in section 4.2.1, the simulated ball is affected by gravity and bouncing force (elastic force). A fixed camera records a 20-frame video of each 4-second simulation to obtain one trajectory (see Figure 13). The ball's size, gravity, and proportion of energy loss per bounce are constant across all trajectories. In this dataset, the color of the ball varies by trajectory, rather than a single color. For each trajectory, the ball's colours are uniformly randomly sampled from a continuous colour space.

We set z c ∈ R 3 with the same counterfactual representation augmentation as in section 4.2 ( S ∈ G ∼ = ( R 2 , +) × SO(2) , K =4). Two of its dimensions are intended to span the horizontal plane and the third unaugmented latent dimension is intended to encode the vertical height. We set z s ∈ R 2 which is intended to represent the ball's colour space.

A.5.2 Result on space-colour disentanglement

Similar to section A.4.2, we evaluate the space-colour disentanglement by augmenting the colour (i.e., style) of the bouncing balls while keeping their locations, and then measure the effects on the normalised z c and z s . Again, a good disentanglement should lead to a change in z s much more significant than z c . Here, we compare two approaches: 1) our model (SPS+) and 2) our model ablating splitting for z s (SPS with z ∈ R 5 and S ∈ G ∼ = ( R 2 , +) × SO(2) ). Note that the ablation model does not differently constrain z 2 (corresponding to the y -axis) than z s . To ensure a meaningful comparison, under colour augmentation, we consider z 2 to be a part of z s of the ablation model and a part of z c of the complete model.

Figure 14 presents the changes in normalised z c and z s measured by L2 distance when we change the colour of an anchor ball whose location is (0, 1, 5) and rendered using white colour. Table 9

provides a more quantitative version by aggregating sampled colour combinations and location pairs. Both results show that our model produces a smaller relative change in z c under timbre augmentation, demonstrating a successful pitch-timbre disentanglement outperforming the ablation model. Note that for the ablation model, z c varies heavily under colour augmentation. Table 8 shows the precision scores of the SPS+ and its ablation against their corresponding random selection for the ball task. These results agree with section A.4.2 and again indicate that the design of an invariant style factor helps with disentanglement.

Blue

Green

Pink

Red

Yellow

0

1

2

SPS

VAE

+ (Ours)

Blue

Green

Pink

Red

Yellow

0

1

SPS

VAE

(Ablation)

||

∆

z

c

||

2

||

∆

z

s

||

2

Figure 15 evaluates the learned colour factor of our model. Each pixel shows the colour of the ball synthesised by the decoder using different z coordinates. The ball colour is detected using naive saturation maxima. In the central subplot, the location factor z 1:3 stays at zeros while the colour factor z 4:5 is controlled by the subplot's x, y axes. As shown in the central subplot, our model (a) learns a natural 2D colour space. The surrounding subplots keep the colour factor z 4:5 unchanged, and the location factor z 1 , 3 is controlled by the subplot's x, y axes. A black cross marks the point where the entire z 1:5 is equal to the corresponding black cross in the central subplot. As is shown by the surrounding subplots, varying the location factor does not affect the colour produced by our model (a), so the disentanglement is successful. The luminosity changes because the scene is lit by a point light source, making the ball location affect the surface shadow. On the other hand, β -VAE (b) learns an uninterpretable colour factor.

z

3

z

1

z

3

z

1

z

3

z

1

z

3

z

1

z

5

z

4

z

3

z

1

z

3

z

1

z

3

z

1

z

3

z

1

(a) SPS

VAE

+,

K

=4 (Ours)

z

3

z

1

z

3

z

1

z

3

z

1

z

3

z

1

z

5

z

4

z

3

z

1

z

3

z

1

z

3

z

1

z

3

z

1

(b)

β

-VAE (Baseline)

×

×

x

×

×

×

×

X

X

X

×

X

×

z

1

z

2

2

σ

z

3

0

+2

σ

(a) SPS

VAE

+,

K

=4 (Ours)

z

1

z

2

2

σ

z

3

0

+2

σ

(b) SPS

VAE

+,

K

=0 (Ablation)

z

1

z

2

2

σ

z

3

0

+2

σ

(c)

β

-VAE (Baseline)

̸

-

-

-

A.5.3 Results on interpretable 3D representation

Figure 16 illustrates the interpretability of learned content factor using latent space traversal. Each row varies only one dimension of the learned 3D content factor, keeping the other two dimensions at zero. Figure 16(a) shows the results of our model. We clearly observe that: i) increasing z 1 (the first dimension of z c ) mostly moves the ball from left to right, increasing z 2 moves the ball from bottom to top, and increasing z 3 mostly moves the ball from far to near. Figure 16(b) is the ablation model without physical symmetry, and (c) shows the result of our baseline model β -VAE, which is trained to reconstruct static images of a single colour (green). Neither (b) nor (c) learns an interpretable latent space.

Table 10 quantitatively evaluates the linearity of the learned location factor. We fit a linear regression from z c to the true 3D location over the test set and then compute the Mean Square Errors (MSEs). A smaller MSE indicates a better fit. All three methods (as used in Figure 16) are evaluated on a single-colour (green) test set. Results show that our model achieves the best linearity in the learned latent factors, which aligns with our observations in Figure 16.

A.5.4 Reconstruction and prior prediction

Similar to section A.4.4, we show that our model suffers little decrease in reconstruction and prediction performance while surpassing the ablation model in terms of L prior by table 11.

A.6 More complicated tasks

The main part of this paper focuses on simple, straight-forward experiments. Still, we supplement our findings by reporting our current implementation's performance on more complicated tasks involving natural melody and real-world video data.

A.6.1 Learning interpretable pitch factors from natural melodies

We report the performance of SPS+ on learning interpretable pitch factors from monophonic melodies under a more realistic setup. We utilize the melodies from the Nottingham Dataset [Foxley, 2011], a collection of 1200 American and British folk songs. For simplicity, we quantise the MIDI melodies by eighth notes, replace rests with sustains and break down sustains into individual notes. We synthesise each non-overlapping 4-bar segment with the accordion soundfonts in FluidR3 GM [Wen, 2013], resulting in around 5000 audio clips, each of 64 steps.

This task is more realistic than the audio task described in A.4 since we use a large set of natural melodies instead of one specified melody line. The task is also more challenging as the prior model has to predict long and more complex melodies. To account for this challenge, we use a GRU [Cho et al. , 2014] with 2 layers of 512 hidden units as the prior model. We perform early-stopping after around 9000 iterations based on spectrogram reconstruction loss on the training set. The model and training setup is otherwise the same as in A.4.

Following A.4.3, We evaluate our approach on notes synthesised with all instruments in GeneralUser GS v1.471 [Chris, 2017] in the MIDI pitch range of C4 to C6, where most of the melodies in Foxley [2011] take place. Note that this is a challenging zero-shot scenario since the model is trained on only one instrument. We compare our model, our model ablating the symmetry loss and a β -VAE baseline. We visualise the embedded z pitch and synthesised pitches for different instruments in Figure 17. Following 12, R 2 results are shown in Figure 18 and Table 12. Even when tested on unseen timbres, our model can learn linear and interpretable pitch factors and demonstrates better embedding and synthesis performance compared with the ablation model, which outperforms the β -VAE baseline.

C4

C6

Pitch

-1

0

1

z

pitch

Embedding

C4

C6

Pitch

-1

0

1

C4

C6

Pitch

-1

0

1

Accordion

Saxophone

Clarinet

Trumpet

Guitar

Violin

Piano

Flute

-

1

0

1

z

pitch

C4

C6

Synthesis

-

1

0

1

z

pitch

C4

C6

-

1

0

1

z

pitch

C4

C6

SPS

VAE

+,

K

=0 (Ablation)

SPS

VAE

+,

K

=4 (Ours)

β

-VAE (Baseline)

0

.

0

0

.

5

1

.

0

R

2

Embedding

SPS

VAE

+

,

K

=4 (Ours)

SPS

VAE

+

,

K

=0 (Ablation)

β

-VAE (Baseline)

Accordion

Piano

Accordion

Clarinet

Flute

Guitar

Saxophone

Trumpet

Violin

0

.

0

0

.

5

1

.

0

R

2

Synthesis

A.6.2 Learning an intepretable location factor from KITTI-Masks

In this task, we evaluate our method's capability on a real-world dataset, KITTI-Masks [Klindt et al. , 2021]. The dataset provides three labels for each image: X and Y for the mask's 2D coordinate, and AR for the pixel-wise area of the mask. Based on the provided labels, we use simple geometrical relation to estimate the person-to-camera distance d , computed as d = 1 / tan( α √ AR ) , where α is a constant describing the typical camera's Field of View (FoV).

We use a 3-dimensional latent code for all models. For SPS, all 3 dimensions are content factors z c and no style factor z s is used. We apply group assumption ( R 3 , +) to augment representations with K = 1 . To measure the interpretability, we fit a linear regression from z c to the ground truth labels and calculate MSEs in the same way as in section A.5.3. The results are shown in Table 13. Linear proj. MSE 1 measures the errors of linear regression from z c to the original dataset labels. Linear proj. MSE 2 measures the errors of linear regression from z c to the person's 3-D location, estimated from the labels.

±

±

±

As is shown in Table 13, MSE 2 is smaller than MSE 1 for SPS, indicating that SPS learns more fundamental factors (person's location) rather than superficial features (pixel-wise location and area). For the baseline methods, MSE 2 is almost equal to MSE 1, and both of them are higher than those of SPS. In summary, our experiment shows that SPS learns more interpretable representations than the baseline (as well as the ablation method) on KITTI-Masks dataset.

z

1

z

2

2

.

0

z

3

0

.

0

2

.

0

(a) SPS

VAE

,

K

=4 (Ours)

2

.

0

0

.

0

2

.

0

(b) SPS

VAE

,

K

=0 (Ablation)

2

.

0

0

.

0

2

.

0

(c)

β

-VAE (Baseline)

̸

-

-

-

Figure 19 shows the generated images, which illustrates that the factors learned by SPS are more linear than those learned by other methods in the human location attribute. For the experiment, We choose all sequences with length ≥ 12 from KITTI-Masks as our dataset; we use 1058 sequences for training and 320 sequences for evaluation; In the inference stage, only the first 4 frames are given. All three methods are trained 30 times with different random initializations. Table 13 shows the average results evaluated on the same test set with 30 different seeds.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to interpretable representation learning based on ''physical symmetry''. The core idea is to learn to predict the temporal evolution of a latent variable which is additionally encouraged to be equivariant under some transformation (e.g. translation or rotation). An autoencoder-like model is designed, where an input sequence is encoded to a sequence of latent variables. From these, three objectives are defined: direct reconstruction, reconstruction after predicting the next state, and an objective involving the symmetric transform. Experiments are presented on an audio-related task (predicting pitches from a sequence of single-note input spectrograms) and a vision task (predicting the coordinates of a moving ball). Further experiments are performed to demonstrate a reduced need for training examples, interpretability of the learned representations, and style-content disentanglement.\n\nSTRENGTHS:\nThe main idea at the core of this work is interesting and original. This idea might also stimulate future work using such physical symmetry constraints. \n\nThe paper provides a variety of experiments, which seem sensibly designed. It is welcome to see experiments dedicated to two different domains (vision and audio). I especially liked the experiments on style-content disentanglement in the Appendix (SPS+). These contain much more realistic scenarios than the main text (albeit still on synthetic datasets).\n\nThe writing in the later sections of the paper is mostly clear and the figures throughout the manuscript are well done. More on the writing below.\n\nThe proposed approach shows only slight or moderate improvements compared to using no physical symmetry constraints. Nevertheless, the idea itself could be considered significant enough to be shared with the community.\n\nWEAKNESSES:\nIn my view, the paper suffers from three main weaknesses:\n\n- The writing in the initial sections of the paper (even the title!) is unnecessarily convoluted and confusing. Initially, I could not understand the main idea of the approach until reading the results sections and the appendix.\n- The experiments are performed on very constrained, synthetic datasets. The train-test split of the audio experiment, especially, seems very simple (the test set contains the same sequences as the training set, just very slightly detuned). For example, based on these experiments, I do not believe the proposed model would work for pitch estimation in-the-wild (whereas SPICE does, to some extent).\n- The results for the proposed approach show a slight or moderate improvement compared to the K=0 baseline. However, most of the observed effect seems to come from the sequence prediction and reconstruction objectives, and not from the proposed physical symmetry (see, e.g., Figure 3, K=0 vs K=4). This is somewhat disappointing. I also believe this fact is not openly discussed in the text.\n\nSome more comments regarding writing:\n- From the title, it is not clear that the approach is applied to the music or image domain. Neither title nor abstract clarify that the proposed approach relies on time-series data. \n- The abstract, intro and conclusion suggest that using (a) using physical symmetry for learning and (b) representation augmentation are two different contributions of the paper, when in fact, they refer to the same thing (i.e., the proposed idea).\n- The term ''prior model'' is used in the abstract, but is not properly explained. In the paragraph starting on line 37, this finally gets explained, but the exposition is again quite convoluted. This would be much easier to understand if the applications used for the experiments (pitch sequences/moving ball) were introduced right at the start.\n- The same goes for ''global invariant style code'', the meaning of which is completely unclear until one reads the appendix. Again, the discussion should be much more concrete and clearly mention what style and content mean in the different application scenarios.\n- Section 2 (Intuition) is actively unhelpful. Instead of a clean explanation (using example applications!), we are given pompous and unscientific prose such as ''we regard symmetry in physical law as a general inductive bias of the human mind'' (line 74).\n- Line 87 introduces the term ''linear pitch factor'', without clarifying the difference between that and simply the term ''pitch''. I suppose the term pitch is not used since the learned latent variables do not encode pitch on a natural (e.g. MIDI) scale, but this only becomes clear to the reader much later in the manuscript.\n- In lines 110f, the reader suddenly learns that the model is a variational autoencoder! This should have been made plain in Section 3.1, or even earlier.\n\nOverall, this is a borderline paper for me, with a tendency towards reject.\n\nQUESTIONS:\n- How important is L_rec, relative to L_prior? That is, can we turn off L_rec and still get similar results?\n- In Equation 2 (and other equations), how do you use the binary cross entropy loss to compare spectrograms (or pictures)? This is very unusual, since spectrograms do not have a value range [0,1]. Is this applied in an element-wise fashion? \n- Line 122: ''In other words, we add or subtract the latent codes by a random scalar.'' -- Unclear. Do you mean you add a scalar *to* the latent code?\n- Line 137f: Is the choice of sequence (a scale upward and downward) relevant? Could you also choose a stationary sequence? Or a random sequence (transposed to different keys)?\n- Lines 286ff: I think the limited usefulness of the VAE compared to the AE stems from the fact that the datasets used are so simple. There are simply no sources of variability that would need to be modeled by the Gaussian prior.\n\nLIMITATIONS:\nThe limitations of the synthetic experimental setups should be discussed more openly in the text.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new way of training a self-supervised system that includes a data augmentation module in the latent space that leverages physical symmetry. This introduction of a transformation covariance in the latent space while training helps learn interpretable, robust and data-efficient representation. An evaluation is performed on synthetic data in two domains: music and videos. The evaluation shows that the new training system is able to learn a presentation that fits human perception.\n\nSTRENGTHS:\n1) The paper is very well written and very interesting\n2) The idea of leveraging symmetry by enforcing latent space covariance seems quite novel to me and could have an impact in the field of Self-Supervised learning.\n3) The method seems to perform as claimed on the presented evaluation (on toy datasets).\n\nWEAKNESSES:\n1) The evaluation is done only on toy problems with synthetic datasets.\n2) Part of the evaluation is not very convincing for several reasons:\n   a) First experiment:\n- The task is too easy for the reported metrics: the R2 scores are saturated to 1, which makes the comparison with the baseline (SPICE) quite questionable.\n- The ablation study (K=0) is a bit disappointing, as the improvement of using symmetry (K=4) doesn't seem significant. So the good properties of the learned representation seem more linked to the encoder/decoder architecture than to the use of symmetry covariance.\n   b) sample efficiency: the correlation between K and sample efficiency is quite unclear in figure 6: for 512 and 2048 samples, K=4 and K=16 seem to have very similar performances.\n3) It's not trivial how the dimension of the latent space and the symmetry should be chosen for problems a bit more complex than the toy ones presented in the evaluation.\n\nQUESTIONS:\nI really don't get why physical symmetry could be seen as a counterfactual inductive bias. Any latent representation model could be used to answer \"what if\" questions about the model (or rather the latent representation itself), which doesn't make them counterfactual inductive.\nCould the author be more specific about what they mean here?\n\nLIMITATIONS:\n1) The link between the group of symmetry to be used for training and the actual task may not be straightforward for tasks a bit more elaborated than the ones proposed in the paper.\n\n2) The claimed interpretability and the link with causality are questionable:\n\n   a) Sample efficiency and extendability are very barely linked to interpretability\n\n   b) The \"interpretability\" of the latent representation in the experimental part is largely due to design choices on the latent space and the symmetry, which can be considered as domain-knowledge priors. So I don't think the model helps generate interpretable representations but is only a way of encoding domain-knowledge priors. For instance, when the authors claim, \"our model learns a more interpretable pitch than...\", the model is actually not learning a more interpretable representation, but rather the prior was correctly handled by the symmetry. Same when they claim \"... leads to a better enforcement of interpretability\". It would be probably safer to talk about \"better enforcement of the prior\".\nThe example with the video has similar issues: implicitly, the chosen symmetry considers that the complicated dimension (height), which as a non-linear behavior, should be kept free, while the two others can be easily contained. This really sounds like a domain knowledge prior.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper utilizes the concept of physical symmetry as a self-supervised constraint within an auto-encoder framework to enhance the learning of interpretable and disentangled representations. The authors validate their approach through experiments conducted on unlabelled monophonic music audio and monocular videos featuring a bouncing ball. The results demonstrate high projection accuracy and successful disentanglement of style and content. Additionally, the paper introduces representation augmentation techniques that improve sample efficiency and enhance interpretability.\n\nSTRENGTHS:\n1. This paper is well-written and easy to understand, demonstrating a cohesive and structured presentation of the research.\n2. Convincing experiment results, which align with the stated objectives, provide strong support for the findings of this study.\n3. The representation augmentation technique is novel and supported by empirical evidence showcasing its effectiveness.\n4. The results on color-texture disentanglement is clear and interpretable\n\nWEAKNESSES:\n1. The experimental setup seems to be somewhat simplistic despite the experimental results being good\n2. Only few experiments on real-world dataset are provided, which is not able to fully demonstrate the generalizability and applicability of your method.\n3. There is a lack of theoretical evidence of your proposed method and further explanation of the relationship between physical symmetry and representation learning.\n\nQUESTIONS:\n1. Could you explain what a random-pooling layer does and the details of how you split the style and content vector? Have you ever tried other methods to disentangle the style and content part?\n2. Will the sequence length (the value of T) affect the final result?\n3. Could you describe any unique aspects or challenges of the dataset you have proposed?\n\nLIMITATIONS:\n1. The generalizability and applicability for real-world dataset may be limited.\n2. The proposed dataset should be more challenging for further utilization on research", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a symmetry equivariance constraints on the transition dynamics of latent representations for time-indexed data. The claim is that imposing these certain symmetries create interpretable model representations that correspond to popular domain-specific representations in the audio and video domains.\n\nExperiments on audio demonstrate how enforcing a translational symmetry recovers an interpretable one-dimensional latent factor corresponding to pitch. Experiments on video show how enforcing translational and rotational symmetries recovers interpretable three-dimensional latent factors corresponding to spatial coordinates. Ablations also show some robustness of learning to mis-specification of symmetries.\n\nSTRENGTHS:\nThis is an inspiring paper! The paper is well-motivated and well-written. The methodology is clearly described and easy to follow (although I do think there might be an even better probabilistic formulation/presentation of the training objective; see the Questions section). The methods make sense and I feel confident that I could implement these ideas myself, based on the description in the paper.\n\nThe experiments are well-executed and support the hypothesis that physical symmetries can provide a powerful inductive bias, at a higher level of abstraction than more domain-specific approaches.\n\nWEAKNESSES:\nMy only major criticism is that the experiments are in somewhat \"toy\" settings. It would be interesting to see an application of these ideas to a more significant problem, with stronger baselines.\n\nA very minor criticism of the (otherwise excellent!) introduction: \"Such an approach is very different from human learning; even without formal music training, one can at least perceive pitch, a fundamental music concept, from the experience of listening to music.\" I think this claim is too strong. It is not clear how much of human perception is learned from experience, vs. baked in to our genetics and brain structure. This claim isn't important to your central argument, so I suggest moderating it a bit.\n\nQUESTIONS:\nIs it possible to frame the training objective (Equation 1) as a proper probabilistic loss? See [1] (the static case) and [2] (the time-indexed case) for the probabilistic formulations of the Gaussian prior. If this is possible, it might help to clarify why/whether it is possible to drop the KL regularization term.\n\nDoes this work relate to previous work on imposing symmetries on intermediate layers of neural networks? E.g. [3] and [4]. I am not deeply familiar with that line of work, but I'm bringing it up because it might be relevant (and at the very least, you might find it interesting if you aren't already aware).\n\n[1] Auto-Encoding Variational Bayes. Diederik P. Kingma, Max Welling.\n\n[2] A Recurrent Latent Variable Model for Sequential Data. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, Yoshua Bengio.\n\n[3] Deep Symmetry Networks. Robert Gens, Pedro Domingos.\n\n[4] Group Equivariant Convolutional Networks. Taco S. Cohen, Max Welling.\n\nLIMITATIONS:\nThe experiments presented in this work use low-dimensional latent spaces: 1 dimension for audio and 3 dimensions for video. It is not clear how to adapt these methods to the high-dimensional latent spaces commonly used (and often required) for expressive models where model performance and capacity is prioritized more highly than interpretability. Furthermore, it is not clear what symmetries we ought to impose upon high-dimensional latent spaces. That said, these questions are clearly beyond the scope of the present work: I see this limitation more as an interesting avenues for future investigation rather than a weakness of the present work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces a new methodology to learn interpretable representations from data by incorporating physical symmetry as a self-consistency constraint in the latent space. It addresses a key challenge in machine learning: how to learn interpretable low-dimensional factors from unlabeled data without relying on domain-specific knowledge. The method, named Self-Supervised learning with Physical Symmetry (SPS), uses the concept of physical symmetry to ensure that the model's learned dynamics are invariant under certain transformations.\n\nThe research applies SPS in two domains: music and computer vision. In music, SPS successfully learns a linear pitch factor from unlabeled monophonic audio, and in computer vision, it learns a 3D Cartesian space from unlabeled videos of a simple moving object. The study suggests that the use of physical symmetry could lead to a new technique called representation augmentation, which enhances the model's sample efficiency.\n\nThe authors draw inspiration from the approach in modern physics where scientists often start from a symmetry assumption to derive laws and predict properties of fundamental particles. This idea of symmetry as a fundamental guiding principle is being used as an \"inductive bias\" for their representation learning model, helping to create an interpretable low-dimensional latent space.\n\nThey view symmetry in physical law not only as a design principle of nature, but also as a cognitive bias of the human mind. This suggests that they believe the learning model should reflect the same intuitive understanding of symmetry.\n\nPhysical symmetry is also used as the basis for a learning technique called \"representation augmentation\". This involves generating additional pairs of training samples from existing ones by applying certain group transformations, which are informed by the symmetries. This process helps to improve sample efficiency and imposes a regularization effect on the learning model.\n\nFinally, through the lens of causality, the authors describe physical symmetry as a counterfactual inductive bias. This means the learning model is designed to ask \"what if\" questions, specifically what would happen if predictions were based on transformed latent codes. This concept also provides constraints on the model's encoder and decoder components, since they are trained end-to-end, keeping the overall structure and function of the model consistent with the underlying symmetry principle.\n\nKey Objectives:\n\nTo introduce and explore the use of physical symmetry as a self-consistency constraint in the latent space of time-series data.\n\nTo develop the SPS methodology that applies physical symmetry to the prior model in an encoder-decoder framework.\n\nTo demonstrate the application of SPS in learning a linear pitch factor from unlabeled monophonic music audio, without any domain-specific knowledge about pitch scales.\n\nTo apply the SPS methodology in the computer vision domain, specifically learning a 3D Cartesian space from unlabeled videos of a simple moving object.\n\nTo examine the desirable properties of SPS, including conciseness, sample efficiency, robustness, and extendability.\n\nTo prove that even with an incorrect symmetry assumption, SPS can still learn more interpretable representations than baseline models.\n\nTo demonstrate the possibility of combining SPS with other learning techniques, allowing for content-style disentanglement from temporal signals.\n\nSTRENGTHS:\nInductive Bias - By leveraging physical symmetry as an inductive bias, the learning model aims to uncover meaningful and interpretable low-dimensional representations from high-dimensional data. This approach aids the model in abstracting critical features of the data, such as the pitch of music notes or the 3D location of a moving object, while filtering out less relevant information.\n\nRegularization and Model Robustness - The method uses symmetry-based representation augmentation and transformations, which generate additional training samples. This augmentation process improves sample efficiency and imposes regularization on the model, reducing overfitting. It also helps in maintaining consistency in predictions, thereby improving model robustness.\n\nThe model is designed with self-consistency in mind. The prior model's predictions for a transformed version of the latent space and the original version should be close, and so should their decoded reconstructions. This self-consistency constraint enhances the regularization of the latent space.\n\nTraining Objectives - The total loss function used in the model is a combination of several loss terms including reconstruction loss, prior prediction loss, symmetry-based loss, and KL divergence loss. This diversified loss function enables the model to optimize different aspects of the learning process, contributing to a better overall model performance.\n\nFlexibility and Generality - This approach is versatile and can be applied to different problems. Different group transformations are used for different problems, demonstrating the method's ability to adapt to different contexts.\n\nVariants - The model has different versions like SPSVAE and SPSAE, providing flexibility based on the specific needs of the task at hand. For instance, SPSAE could be used when a simpler model architecture without the KL divergence term is sufficient.\n\nWEAKNESSES:\nDependence on Correct Symmetry Assumptions: While the experiments show that the method is robust to incorrect symmetry assumptions, choosing the right symmetry transformation S is crucial for optimal results. Incorrect assumptions can still lead to an inferior performance. Real-world data may contain complex symmetries not known a priori, which could lead to difficulty in correctly specifying these assumptions.\n\nDependence on Augmentation Factor: The efficiency and effectiveness of the model are significantly affected by the augmentation factor K. Choosing an optimal K is a challenge and requires extensive experiments. A poorly chosen K could result in less efficient training or lower performance.\n\nHigh-Dimensional Latent Space: While the method has been demonstrated on 1D and 3D latent spaces, its effectiveness in higher-dimensional latent spaces is not fully explored. Real-world data can often have high-dimensional latent spaces, and it may be challenging to maintain the same level of performance in these cases.\n\nOverfitting: While representation augmentation can increase the robustness of the model, applying it excessively might lead to overfitting, especially when data is scarce.\n\nQUESTIONS:\nHow do the authors propose this approach would be scalable for other tasks and what are the ways to prevent overfitting with the approach? \n\nSPS can still learn interpretable representations even with incorrect symmetry assumptions. What might be the theoretical basis for this? Could there be instances where this doesn't hold true?\n\nHow does the proposed SPS model relate to, or differ from, other models or techniques used in representation learning? Could SPS be integrated with other methods to address its limitations?\n\nLIMITATIONS:\nAuthors do inform about the range of limitations for the work \n\nPartial Information Capture: The authors note that when the underlying concept following physical symmetry only encompasses a part of the information present in the time series data and cannot fully reconstruct the inputs, SPS may not work effectively. This issue is potentially related to content-style disentanglement.\n\nInability to Distill Concepts from Multibody Systems: The current model struggles with learning concepts from systems with multiple interacting components. As examples, the authors mention that SPS fails to learn the concept of pitch from polyphonic music, or understand 3D space from videos featuring multiple moving objects.\n\nLack of Formal Theory for Quantification: The authors highlight the need for a formalized theory to quantify the impact of representation augmentation. Such a theory could help measure the degree of freedom in the latent space both with and without physical symmetry, and help explain why even incorrect symmetry assumptions can still lead to correct and interpretable latent space.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}