Learning Basic Interpretable Factors from Temporal Signals via Physics Symmetry
Authors: Xuanjie Liu
Daniel Chin
Yichen Huang
Gus Xia
Venue: Submitted to ICLR 2023 (2023)
Keywords: Physics Symmetry
Time series data
Self-supervised Learning
Representation Augmentation

Abstract: We have recently seen great progress in learning interpretable music representations, ranging from basic factors, such as pitch and timbre, to high-level concepts, such as chord, texture and melody contour. However, most methods rely heavily on music domain knowledge and it remains an open question how to learn interpretable and disentangled representations using inductive biases that are more general. In this study, we use \textit{physical symmetry} as a self-consistency constraint on the latent space. Specifically, it requires the prior model that characterises the dynamics of the latent states to be \textit{equivariant} with respect to a certain group transformation (say, translation and rotation). We show that our model can learn \textit{linear} pitch factor (that agrees with human music perception) as well as pitch-timbre disentanglement from unlabelled monophonic music audio. In addition, the same methodology can be applied to computer vision, learning the 3D Cartesian space as well as space-colour disentanglement from a simple moving object shot by a single fix camera. Furthermore, applying physical symmetry to the prior model naturally leads to \textit{representation augmentation}, a new learning technique which helps improve sample efficiency. 

Under review as a conference paper at ICLR 2023
LEARNING BASIC INTERPRETABLE FACTORS FROM TEMPORAL SIGNALS VIA PHYSICAL SYMMETRY
Anonymous authors Paper under double-blind review
ABSTRACT
We have recently seen great progress in learning interpretable music representa- tions, ranging from basic factors, such as pitch and timbre, to high-level concepts. such as chord andtexture. However, most methods rely heavily on music domain knowledge and remains an open question how to learn interpretable and disen- tangled representations using inductive biases that are more general. In this study, we use physical symmetry as a self-consistency constraint on the latent space. Specifically, it requires the prior model that characterises the dynamics of the la- tent states to be equivariant with respect to a certain group transformation. We show that model can learn linear pitch factor (that agrees with human music perception) as well as pitch-timbre disentanglement from unlabelled monophonic music audio. In addition, the same methodology can be applied to computer vi- sion, learning the 3D Cartesian space as well as space-colour disentanglemen from a simple moving object shot by a single fixed camera. Furthermore. applying physical symmetry to the prior model naturally leads to representation augmenta- tion, new learning technique which helps improve sample efficiency.
INTRODUCTION
Interpretable representation-llariig models have achieved great progress for various types of time- series data. Taking the music domain as an example, tailored deep generative models (Ji et al. 2020) have been developed texture, etc. However, most models still rely heavily on domain-specific knowledge. For example, to use pitch scales or instrument labels for learning pitch and timbre representations (Luo et al. 2020
learn pitch, timbre, melody contour, chord progression, accompaniment
2019 Engel et al. 2020: [in et al. [202 Esling et al. (2018) and to use chords and rhythm labels for learning higher-level representations (Akamal 2019: Yang et al. 20 99: Wang et al. 2020 Wei & Xia 2021). Such approach is very different from human learning; even without formal music training, one can at least perceive basic factors such as pitch and timbre from the experience of listening to music. In other words, it remains open question how to learn interpretable music representations using inductive biases that are more general. We see a similar issue in other domains. For instance, various computer-vision models (McCarthy & Ahmed 2020: Trevithick & Yang 2021 Mescheder eet al. 2019 Riegler et al., 2017) can learn 3D representations of human faces or particular scene by incorporating domain knowledge (e.g., labelling of meshes and voxels. 3D-specific setups such as multi-cameras, 3D convolution, etc.) but it remains a non non-trivial task to trace the 3D location of a simple moving object from monocular videos in a self-supervised fashion.
In this study. we explore to use physical symmetry (i.e., symmetry of physical laws) as weak self-consistency constraint for the learned latent x space. As indicated in Figure [ this general inductive bias requires that after a certain transformation S (e.g., translation rotation) in the latent space, the learned prior model R, which the induced physical law describing the temporal flow of the latent states, should output equivariant predictions. Formally, a++1 R(x,) if and only if 21+1 R(z²). where S = =S().. In other words, R and S are commutable operations for z, ie., R(S(2)) S(R(2)). Note that this approach is fundamentally different from most existing symmetry-informed models (Bronstein et al. 2021), in which the symmetry property is used to constrain the encoder or the decoder.
Specifically, we design self-super learning with physics symmetry (SPS), a method that adopts an encoder-decoder framework and applies physical symmetry to the prior model. We show that

---

Under review as a conference paper at ICLR 2023
E
Xe+1
D
S
R
E
D
S
Figure 1: Physical symmetry, the fundamental inductive bias of this study.
with the right symmetry assumptions, model learns linear pitch factors that agree with human music perception from monophonic music audio, without any domain-specific knowledge about pitch scales or signal-level regularities. If we further assume extra global invariant latent code, the model can learn pitch-timbre disentanglement without instrument labelling. Moreover, we show that the same methodology can be applied the computer vision domain, learning 3D Cartesian space as well as space-colour disentanglement from monocular videos of a bouncing ball shot from fixed perspective.
2 INTUITION
The idea of using physical symmetry for representation learning comes from modern physics. In classical physics. scientists usually first induce physical laws from observations and then discover symmetry properties of the law. (E.g., Newton's law of gravitation, which was induced from plan- etary orbits, is symmetric with respect to Galilean transformation.) scientists often start from symmetry assumption, based on which they derive the corresponding law and predict the properties (representations) of fundamental particles. (E.g., general relativity was developed based on a firm assumption of symmetry with respect to Lorentz transformation).
contrast, in modern physics,
Analogously, we use physical symmetry as an inductive bias our machine learning model, which helps learn regularised prior and an interpretable latent space. In other words, if it is a belief of many physicists that symmetry in physical law is a main design principle the nature, we regard symmetry in physical law major useful inductive bias of the representation learner.
The introduction of physical symmetry to the learned prior model naturally Ieads to representation augmentation, novel learning technique which helps improve sample efficiency. As indicated in Figure [ representation augmentation means to "imagine" 2, as the training sample of the prior model π.. Representation augmentation can be regarded a regularisation of the prior model, since it requires the prediction of the sequence to be equivariant with respect to certain group transformation of S. It also constrains the encoder and decoder indirectly through the prior model since the network is trained in an end-to-end fashion.
METHODOLOGY
Our goal learn a disentangled and interpretable representation z; of each high-dimensional sam- ple from time-series X1.r. The disentanglement of 2; is at two levels. First, is divided into two factors: 24, and 2,c, where 2,, is the global invariant style and ži,e is the content representa- tion that changes over time. More importantly, we aim to further disentangle the spatio-temporal content factor 21,e using physical symmetry such that it equivariant with respect to the prior model and each dimension of it is interpretable and consistent with human perception.
We focus on two specific problems in this paper. The primary problem to learn pitch and timbre factors of music notes from music audio, where each x; is spectrogram of a note. Ideally, ži,e is a ID content factor representing the pitch and 2;,a is a style factor representing the timbre. Another problem is to leam 3D Cartesian location and colour factors of a simple moving object (a bouncing ball) from its trajectory shot by a fixed, single camera. In this case, each æ; is an image. Ideally, ž is learned to be 3D content factor representing the location and zi.s represents the global colour.
2

---

Under review as a conference paper at ICLR 2023
3.1 MODEL
Figure shows model design. During the training process, the temporal data imput X1.7 is first fed into the encoder E obtain the corresponding representation Z Zirr is then split into two parts: the style factor 7ı: and the content factor 71:7,c. The style factor 71:7,s is passed through the random-pooling module P, where one sub-element Σr, is randomly picked. The content factor Zi:T.e fed into three branches. In the first branch (the green line), Zi:7,e combined with 2r,a by concatenating each sub-element ži,e with 2r,s, and fed to the decoder D to reconstruct Xi.T- In the second branch (the orange line), Zi:T.e is passed through the prior model R to predict next step, 22.r+ which is then combined with 2,,, to reconstruct N2.T+1- In the third branch (the blue line), we sequentially transform 71:7,e with S. pass it through R. and transform back using the inverse transformation S 1 to predict another version of the next step 22:T+1,e> which finally combined with 2 to reconstruct X2.T+1- We get three outputs from the model: X1.7 82:T+1 and 82:T+1-
The underlying idea of physical symmetry is that the dynamics of latent content factor and its trans- formed version follow the same physical law characterised by R. Therefore, z and ž should be close to each other and so are and X, assuming is proper transformation. This self-consistency constraint helps the network learn more regularised latent space. In addition, Zr,s further helps disentangle the style factor by assuming global invariant style code over time.
ZI.Tx
Z
Xir
split
Z 1:7
Z1:TT
22. 1.0
R2.T+1
or
S
12.++1
82.T+1
R
12.+ic
25
Figure 2: An overview of our model. X1:T is fed into the encoder E to obtain the corresponding representation Zı:T, which is then split into two parts: the style factor Zi:T, and the content factor Z1.T.c. The style factor is passed through the random-pooling layer P, where a sub-element 2r,, is randomly selected. The content factor is fed into three different branches and combined with 2r,s reconstruct three outputs respectively: X 82.T+1 and X2.T+1. Here, R is the prior model and is the symmetric operation. The inductive bias of physical symmetry enforces 21,T,e to be equivaraint w.r.t. to R, so therefore ž and ž should be close to each other and so are x and X.
3.2 TRAINING OBJECTIVE
The total loss contains four terms: reconstruction loss Lres prior prediction loss Spriee symmetry- based loss Lyyms and KL divergence loss LKLD. Formally,
C Cres + X Sprior + X2hrym + AaLLID.
(1)
where Å1. d2 and Ag are weighting parameters. By referring to the notations in section 3.]
Live LuCE(X Xv.)) + LBCE x2.r) + LBCE S
(2)
Cpror lz(22. Ze:T.e,
(3)
Csym = la(22 (.. + +22 (2 22..)).
(4)
Lastly, the CKLD is the Kulback-Leibler divergence loss between the posterior distribution of 2; and standard Gaussian.

---

Under review as a conference paper at ICLR 2023
3.3 SYMMETRY-BASHE REPRESENTATION AUGMENTATION
We name S with representation augmentation since it creates extra fake sequences of z (i.e., imagi- nary experience based on group assumption) help train the prior. In practice, we apply K differ- ent transformations S to -1, to generate K fake sequences. Thus, the two terms of symmetry- based loss can be specified as:
E2 (22T, 22TT.)
1
Σ
(S
(5)
K
K
82 (Z2:T. (
1
Σ
( (2.T,a),
(6)
K
k=1
where the lower case denotes the index a specific transformation and we refer to K as the augmentation factor. Likewise, the last term of reconstruction loss can be specified as:
K
(
Σ k=1
LECE (R(S5572((((4441111111 2r,)) X2.T).
K
Each S applied to each sequence Li belongs to a certain group, and different groups are used for different problems. For the music problem, we assume Zi be 1D and random S E G = (R, +). In other words, we add or subtract the content factor by random scalar. As for the video problem, we assume Z be to 3D and random E G ~ (RR2+ SO(2). In other words, random rotation and shift are applied on two dimensions of Z..c-
4 RESULTS
We run experiments on two problems. In section 4.1 we present the results of learning pitch content and timbre style from monophonic music. In section 4.2 we present the results of learning 3D- location content and colour style from monocular videos of a bouncing ball. For both problems, we use representation augmentation factor of K learned content is interpretable, and whether the content-style disentanglement is successful. We present additional results in the appendix, including more complicated and realistic data in section A.3 more flexible assumptions on symmetry in section A.2.4 and results of applying SPS on auto- encoders without the variational constraints section
The focus in this section is to see whether the
4.1 LEARNING PITCH AND TIMBRE FACTORS FROM MUSIC AUDIO
4.1.1 DATASET
We synthesise a dataset that contains around 2400 audio clips played by multiple instruments. Each contains 15 notes in major scales with the first half ascending and the second half descending. We refer readers appendix section A.1.3 for details.
4.1.2 RESULTS ON INTERPRETABLE PITCH SPACE
Figure shows that the pitch factor learned by our model has linear relation with the true pitch. Here, we use Zpitch as the synonym of ze to denote the content factor. The plot shows the mappings of two tasks and four models. In the embedding task (the first row), z-axis is the true pitch and y-axis is embedded 2pitch- In the synthesis task (the second row), æ-axis is Zpitch and y-axis is the detected pitch (by YIN algorithm, a standard pitch-estimation method by De Cheveigné Kawaharal )2022) of decoded (synthesised) notes. The fours models involved are: 1) our model, 2) our model without symmetry (i.c., no representation augmentation during training). 3) a 3-V trained to encode single-note spectrograms from a single instrument (banjo) to ID embeddings, and 4) SPICE (Gieller et al.) 2020). a SOTA unsupervised pitch estimator with strong domain knowledge on how pitch linearity is reflected in log-ft frequueyyyy spectrograms. As the figure shows, without explicit knowledge of pitch, our model learns a more interpretable pitch factor than 3- VAE, and the result is comparable SPICE.
4

---

Under review as a conference paper at ICLR 2023
SPS (Ours)
Ours w/o Symmetry
8- VAE (Baseline)
SPICE (Baselinc)
Embedding
2.5
0.75
Spare
0-
0-
0.30
0.25
-5-
C2
C4
C6
C2
C4 Pitch
C6
C2
C4 Pitch
C6i
C2
C1 Pitch
C6
Pitch
Accordion
Guitar
Synthesis
Deteeted Pitch
O6-
C6
Saxophone
Violin
C4
C1
Clarinet
Piano
E. Piana
Flute
C2-
C2
C2
Trumpet
-5
0 -piich
-25
0.0
2.5
0.0
2.5
Sprcch
Spatch
Figure 3: A visualisation of the mapping between the 1D content factor and the true pitch. In the upper row, models encode notes in the test set to Spitch- The z axis shows the true pitch and the y axis shows the learned pitch factor. In the lower row, the x axis traverses the &pitch space, The models decode Epitch to audio clips. We apply YIN to the audio clips to detect the pitch, which is shown by the y axis. In both rows, a linear, noiseless mapping ideal, and our method performs the best. All results evaluated on the test set.
4.1.3 RESULTS ON PITCH-TIMBRE DISENTANGLEMENT
We evaluate the content-style disentanglement using factor-wise data augmentation following Yang et al. (2019). Namely, we change (i.c., augment) the instrument (i.c., style) of notes while keeping their pitch. and then measure the effects on the encoded &, and 2g. We compare the normalised &n and z,, ensuring they have the same dynamic range. Ideally, the change of z, should be much more significant than z.. Here, we compare four approaches: 1) model, 2) model without splitting for 2, (i.c.. our model without the dashed black path shown in Figure 2 but fed 2, together with že into the prior model, 3) GMVAE (Luo al. 2019). domain-specific pitch-timbre disentanglement model trained with explicit pitch labels, and 4) TS-DAAE (Luo et al. 2022), a latest unsupervised pitch-timbrg disentanglement model based on Disentangled Sequential Autoencoder (DSAE).
||
||
SPS (Ours)
Ours w/o 2, splitting
GMVAE (Baseline)
TS-DSAE (Baseline)
5.0
2.5
0.0
Clarinet
E. Piano
Flute
Guilar
Saxophone
Trumpet
Violin
Clarinet
E. Plann
Plute
Guitar
Plano
Saxophone
Trumpet
Vialin
Clarinet
E. Plano
Flute
Guitar
Piano
Saxophone
Trumpet
Violin
Clarinet
E. Piano
Flute
Guitar
Plano
Saxophone
Trumpet
Vialin
Figure 4: Comparisons for Aze and As, for different instruments against accordion, with pitch kept constant at MIDI pitch D3. Az, and Az, are changes in normalised ze and 2,, that higher black bars relative to white bars means better results. All results are evaluated on the test set.
Figure presents the changes in normalised 2, and 2, measured by L2 distance when we change the instrument of an anchor note whose pitch is D3 and synthesised by accordion. Table I provides a more quantitative version by aggregating all possible instrument combinations and all different pitch pairs. Both results show that model produces smaller relative change in z, under timbre
5

---

Under review as a conference paper at ICLR 2023
augmentation, demonstrating successful pitch-timbre disentanglement outperforming both the ab- lation and baseline. Note that for the ablation model, & varies heavily under timbre augmentation, seemingly containing timbre information. This result indicates that the design of an invariant style factor over the temporal flow is necessary to achieve good disentanglement.
Mean ratios of changes in normalised =. and 2, under timbre augmentation across all possible instrument combinations under different constant pitches in the test set.
Table 1:
Methods
|| |2/| Az 4
Ours w/o 2, splitting
2.20
GMVAE (Baseline)
0.67
DS-DSAE (Baseline)
0.65
SPS (Ours)
0.49
4.2
LEARNING LOCATION AND COLOUR FACTORS FROM VIDEOS OF MOVING OBJECT
4.2.1
DATASET
We run physical simulations of bouncing ball in 3D space and generate 4096 trajectories. The ball's colour, initial location, and initial velocity for each trajectory are randomly sampled. Please see appendix A.1. for more details.
4.2.2
RESULT ON INTERPRETABLE 3D REPRESENTATION
(a) SPS (Ours)
(b) Ours w/o Symmetry
(c) 8-VAE (Baseline)
21
72
9g
-20
0.0
2.0
-20
0.0
2.0
2.0
0.0
2.0
Figure 5: Row i shows the generated images when changing and keeping žμ: 0, where the axis varies z; from -2 to 2. In (a), changing 22 controls the ball's height, and changing 2 Ry moves the ball parallel the ground plane.
Figure illustrates the interpretability of learned content factor using latent space traversal. Each row varies only one dimension of the learned 3D content factor, keeping the other dimensions at zero. Figure 5 a) shows the results of our model. We clearly observe that: i) increasing 21 (the first dimension of 2e mostly moves the ball from left to right, increasing 22 moves the ball from bottom mostly moves the ball from far to near. Figure (((b) is the ablation model without physical symmetry, and (c) shows the result of our baseline model 3-VAE, which trained to reconstruct static images of a single colour (green). Neither (b) (c) learns interpretable latent space.
to top, and increasing
Table 2 quantitativel evaluates the linearity of the learned location factor. We fit linear regression from 2. to the true 3D location over the test set and then compute the Mean Square Errors (MSEs). A smaller MSE indicates better fit. All three methods (as used in Figure (5) evaluated single-colour (green) test set. Results show that our model achieves the best linearity in the learned latent factors, which aligns with observations in Figure

---

Under review as a conference paper at ICLR 2023
Table 2: Linear fits between the location and the learned location factor. We the encoder the test set to obtain data pairs in the form of (location factor, true coordinates). We then run linear fit the data pairs evaluate factor interpretability.
Method
I' axis MSE y axis MSE
axis MSE
MSE 5
3-VAE (Baseline)
0.37
0.76
0.73
0.62
Ours w/o Symmetry
0.35
0.72
0.68
0.58
SPS (Ours)
0.11
0.06
0.09
0.09
4.2.3 RESULT SPACE-COLOUR DISENTANGLEMENT
Similar to section 4.1.3 we evaluate the space-colour disentanglement by augmenting the colour (i.e.. style) of the bouncing balls while keeping their locations, and then measure the effects on the normalised & and 2.. Again, good disentanglement should lead a change in 2, much more significant than splitting for 2s- The ablation model follows the same configuration as in section 4.1.3]
Here, we compare two approaches: 1) our model and 2) our model ablating
subtle difference is that the ablation model does not differently constrain 22 (corresponding to the y-axis)
than Za- To ensure a meaningful comparison, under colour augmentation, we consider 22 to be a part of 2g of the ablation model and part of of the complete model.
|A
| la
SPS (Ours)
Ours w/o 2s splitting
Blue
Groen
Pink
Yellow
Blue
Cyan
Groen
Pink
Red
Yellow
Figure 6: Comparisons of normalised Aze and Az, for different colours against white, with the ball's location kept constant at (0, 1, 5). Higher black bars (relative white bars) means better result. (Results are evaluated on the test set.)
Table Mean ratios of changes in normalised 2e and 2, under colour augmentation across sampled colour combinations keeping locations constant. Results evaluated on the test set.
Methods
||Azellz/ /A2ellz/||A,,
Ours w/o n, splitting
1.62
SPS (Ours)
0.54
F presents the changes in normalised 2, and E, measured by 1.2 distance when we change the instrument of an anchor ball whose location is (0, 1) and rendered using white colour. Table vides a more quantitative version by aggregating sampled colour combinations and location pairs. Both results show that our model produces smaller relative change in Re under timbre augmen- tation, demonstrating successful pitch-timin timbreibee disentanglement outperforming the ablation model. Note that for the ablation model, 2e varies heavily under colour augmentation. This result agrees with section 4.1.3 and again indicates that the design of an invariant style factor helps with disen- tanglement.

---

Under review as a conference paper at ICLR 2023
5 ANALYSIS
To better understand how representation augmentation leads to an interpretable content factor, we train a simplified version of our model with style factor on the videos of single-colour (green) bouncing ball. choose the vision problem since 3D content manifests a more obvious difference when physical symmetry applied.
5.1 REPRESENTATION AUGMENTATION IMPROVES SAMPLE EFFICIENCY
Figure shows that larger factor of representation augmentation leads lower linear projection loss (the measurement introduced in section 4.2.2) of the learned 3D representation. Here, K is the augmentation factor, and K means the model trained without physical symmetry. The comparative study is conducted on 4 training set sizes (256, 512, 1024 and 2048), in which each box plot shows the results of 10 experiments trained with fixed K and random initialisation. We see that a larger K leads to better results and compensates for the lack of training data. E.g., the loss trained on 256 samples with K = is comparable to the loss trained on 1024 samples with K = O, and the loss trained on 512 samples with K 4 is even lower than the loss trained 2048 samples with K = In other words, when K = O, increasing the number of training samples beyond a certain point does not further shrink the error, but increasing K still helps. In appendix [A.2.4] further show that even with incorrect group assumptions, our method still helps learn interpretable content factors.
256 Samples
512 Samples
1024 Samples
2048 Samples
Libosk jara]. loos
0.6
0.2
16
166
10
1.6
Figure 7: Linear projection MSE for different augmentation factors (K) and training set sizes. Rep- resentation augmentation improves sample efficiency as smaller values mean better results.
5.2 REPRESENTATION AUGMENTATION IMPROVES DISENTANGLEMENT
Figure 8 visualises the latent space during different stages of model training, and we see that larger K leads to better disentanglement in an earlier stage of training. The horizontal axis shows the training epoch. Three experiments with different K values (x0, x4, x16) are stacked vertically. Each experiment is trained twice with random initialisation. Each subplot shows the orthogonal projection of the 2e space onto the plane spanned by and 23, therefore hiding most of the y- axis (i.e. ball height) wherever disentanglement is successful. During training, the role of physical symmetry is "straighten" the encoded grid and larger K yields stronger effect.
RELATED WORK
The idea of using predictive model for better self-supervised learning has been well established (Oord et al. 2018 Chung et al., 2013 LeCun 2022). In terms of model architecture, our model belongs to the family of disentangled sequential autoencoders (Bai et al., 2021 Hsu et al. 2017 Vowels et al. 2021 Yingzhen & Mandi 2018 Zhu et al. 2020). It is very similar to VRNN CCung et al. 2015) we remove the global invariant part from the overall network. In addition, our model be
variation of embedddnn predictive architecture (JEPA) in LeCun (022) if eliminate the reconstruction losses on the observation. In fact, we see the network topology model the "hardware" and the learning strategy (e.g., contrastive method, regularised method,
8

---

Under review as a conference paper at ICLR 2023
Ours w/o Symmetry
Ours, Representation Augmented by
Ours, Representation A Augmented by 16x
2-
4
2
2
22
-2
2
-2
-2
-2
2
-2
2
-2
Epoch
331
669
1831
1060
2000
2331
2609
3000
3331
3009
4331
4000
5000
Figure 8: A visualisation of the learned latent space against training epoch. We plot how the encoder projects an equidistant 3D grid of true Cartesian coordinates onto the z space. Different colours denote respective axes in the true coordinates.
or a mixed one), as "software". The main contribution of this study lies in the learning strategy to use physical symmetry to limit the capacity of the latent space, and to use representation augmentation to increase sample efficiency.
The notion of "symmetry" and the use of symmetry for disentanglement are also not new in repre- sentation learning (Higgins et al. 22018: Bronstein et al. (021). Most symmetric- based methods care about the relation between observation and latent (Sanghi 2020 Quessard et al. 2020 Dupont et al. 2020 Huang et al. 2021). E.g.. when a certain transformation is applied to x. 2 should sim- ply keep invariant or follow same/similar transformation. Such assumption inevitably requires some knowledge in the domain of x. In contrast, the physical symmetry used in this study focuses solely the dynamics of z, and therefore we only have to make assumptions about the underlying group transformation in the latent space. We see two most relevant works in the field of reinforce- ment learning (Mondal al., 2022 Dupont al., 2020), which applied an equivariant assumption very similar to the physical symmetry used in this paper. The major differences are twofold. First, disentangle the basic factors, our method requires no interactions with the environment. Second, our method is much more concise; needs no other tailored components or other inductive biases such as symmetric embeddings network and contrastive loss used in Dupont et al. (2020) or MDP homomorphism applied in Mondal et al. (2022).
CONCLUSION
In this paper, we propose a methodology that uses physical symmetry to learn disentangled and interpretable representations from time-series data. Experiments show that physical symmetry ef- fectively regularises the latent space by enforcing the prior model to be equivariant with respect to group transformations. Under a proper group assumption, our method learns an interpretable 1D pitch factor (that agrees with human music perception) as well as a good pitch-timbre disentangle- ment from music audios without any labels pitches or instruments. We also show that with the same method, we can learn interpretable 3D Cartesian location factor well as good location- colour disentanglement from monocular videos bouncing ball shot from a fixed perspective. In addition, training technique, representation augmentation, is developed to couple with phys- ical symmetry. Analysis shows that representation augmentation leads to better sample efficiency and representation disentanglement.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The source code for training and testing the models, as well as generating the figures and tables, is publicly available at http://hww...ooooommoooo//ooo/oa from-temporal signals signals-via-phsseen sla-physccll--mmmerr
ACKNOWLEDGMENTS
Redacted for review.
REFERENCES
Taketo Akama. Controlling symbolic music generation based on concept learning from domain knowledge. In Arthur Flexer, Geoffroy Peeters, Julián Urbano, and Anja Volk (eds.). Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pp- 816-823, 2019. URL
pdf
Junwen Bai, Weiran Wang, and Carla P Gomes. Contrastively disentangled sequential variational autoencoder. Advances Neural Information Processing Systems, 34: 34:10105-10118, 2021. 2021.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in Neural Information Processing Systems, 28, 2015.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics. and gauges. arXiv preprint arXiv:2104.13478, 2021.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics. doi:
URL httthtts::/ http:///aaaatoogg.oorrr.o org/214441412
Collins Chris. GeneralUser GS, 2017.
URL
com/
genera luser.ppp)
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Sys- fems, volume 28. Curran Associates, Inc., 2015. URL https:htt:////rooeeeet
ngs nwwriiurip.... CC722e/000M111111111000332222222 Paweee.-2pppprrpppee pdf
Alain De Cheveigné and Hideki Kawahara. Yin, fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, 111(4):1917-1930, 2002.
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference Machine Learning, pp. 2761-2770. PMLR, 2020.
Jesse Engel, Rigel Swavely, Lamtharn Hanoi Hantrakul, Adam Roberts, and Curtis Hawthorne. Self-supervised pitch detection by inverse audio synthesis. In ICML 2020 Workshop on Self- supervision in Audio and Speech, 2020. URL httpss:/openzevvew. wwteeerrvieennnee/eeeee wwwp:/oumtt....c..mmmmm
RIVTYWhsky7,
Philippe Esling, Axel Chemla-Romen Santos, and Adrien Bitton. Bridging audio analysis, per- ception and synthesis with perceptually-reguleee variational timbre spaces. In Proceedings of the 19th International Society for Music Information Retrieval Conference, ISMIR 2018, pp-
175-181, 2018
Eric Foxley. Nottingham database,
2011.
URL https://ifdo. ca/ cww.:/cco//seeur////ee seymour
nott iottingham/noke h nngham....mmmmm.mmmm.mmmm...mmm. html
10

---

Under review as a conference paper at ICLR 2023
Beat Gfeller, Christian Frank, Dominik Roblek, Matt Sharifi, Marco Tagliasacchi, and Mihajlo Ve- limirović. Spice: Self-supervised pitch estimation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 18-81288 2020.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.0222300 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in Neural Information Processing Systems, 30, 2017.
Siyuan Huang, Yichen Xie, Kong-Chun Zhu, and Yixin Zhu. Spatio-temporal self-supervised repre- sentation learning for 3d point clouds. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 6535-6545 2021.
Shulei Ji, Jing Luo, and Xinyu Yang. A comprehensive survey on deep music generation: Multi-level representations, algorithms,, evaluations, and future directions. arXiv preprint (.Ni::001100680) 2020.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with tempo- ral sparse coding. In International Conference on Learning Representations, 2021. URL
https: pettps://openee /ttp:///ppeeeeeieeeemm net http://ereeeeeeeeeeeeooorrrrr/ee fow.:
Yann LeCun. A path towards autonomous machine intelligence. preprint posted on openreview, 2022.
Liwei Lin, Gus Xia, Qiuqiang Kong, and Junyan Jiang. A unified model for zero-shot music source separation, transcription and synthesis. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranenburg, and Ajay Srinivasamurthy (eds.), Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, pp. 381-388, 2021. ISBN 978-1 778-1-7327299-0022 URL https: /attpp://arrcives.. iativve.sssnne ismir nnt// ismi httpr/222/pappee/0000.. pdf
Yin-Jyun Luo, Kat Agres, and Dorien Herremans. Learning disentangled representations of timbre and pitch for musical instrument sounds using gaussian mixture variational autoencoders. Pro- ceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019.
Yin-Jyun Luo, Kin Wai Cheuk, Tomoyasu Nakano, Masataka Goto, and Dorien Herremans. Unsu- pervised disentanglement pitch and timbre for isolated musical instrument sounds. In Proceed- ings of the 2 21st International Society for Music Information Retrieval Conference, ISMIR 2020, pp. 700-707, 2020.
Yin-Jyun Luo, Sebastian Ewert, and Simon Dixon. Towards robust unsupervised disentanglement of sequential data case study using music audio. In Lud De Raedt (ed.). Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. IJCAI-22, pp. 3299-3305. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10.24963/ jjcai.2022/455 /tthttss:::/// //tps////oo..ee/ http://ooo..o///////// org/10....... 2 4943/115caaa 2022 / Main Track.
Ollie McCarthy and Zohaib Ahmed. Hooligan: Robust, high quality neural vocoding. arXiv preprint arXiv:2008. 02493, 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 14600-447,, 2019.
Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh, Eqr: Equivariant representations for data-efficient reinforcement learning. In International Conference on Machine Learning, pp. 15908-15926. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint / 807003488 2018

---

Under review as a conference paper at ICLR 2023
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33:19727-19737. 2020.
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representa- tions at high resolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3577-3586, 2017.
Aditya Sanghi. Info3d: Representation learning on 3d objects using mutual information maxi- mization and contrastive learning. In European Conference on Computer Vision, pp. 626-642. Springer, 2020.
Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15182-15192, 2021.
Matthew Vowels, Necati Cihan Camgoz, and Richard Bowden. Vdsm: Unsupervised video dis- entanglement with state-space modeling and deep mixtures of experts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8176-8186, 2021.
Ziyu Wang. Dingsu Wang, Yixiao Zhang, and Gus Xia. Learning interpretable representation for controllable polyphonic music generation. In Julie Cumming, Jin Ha Lee. Brian McFee, Markus Schedl, Johanna Devaney, Cory McKay, Eva Zangerle, and Timothy de Reuse (eds.). Proceed- ings of the 21th International Society for Music Information Retrieval Conference, ISMIR 2020, pp. 662 669, 2020. ISBN 978-0-9813537-1-5. URL http: archives ismir. .net/ ssmi wwt.0//aaee////////
pd
Shiqi Wei and Gus Xia. Learning long-term music representations via hierarchical contextual con- straints. In Jin Ha Lee, Alexander Lerch, Zhiyao Duan, Juhan Nam, Preeti Rao, Peter van Kranen- burg, and Ajay Srinivasamurthy (eds.). Proceedings of the 22nd International Society for Music Information Retrieval Conference. ISMIR 2021, pp. 738 745, 2021. ISBN 978-1-7327299-0-2 URL httts:://acchhht
sm net. sm 2021 /ttp:////////////00/02000000 .df
Frank Wen. The fluid release 3 general-MIDI soundfont, 2013. URL https: /mmbbmmm.rr...ee..eemmmmmm..er..
keymus cian, www.///////////oocc FulierrrFFuudd3 wwwdduuddc_c GM/ Btt.MM//iddx.... html.
Ruihan Yang, Dingsu Wang. Ziyu Wang, Tianyao Chen, Junyan Jiang, and Gus Xia. Deep music analogy via latent representation disentanglement. In Arthur Flexer, Geoffroy Peeters, Julián Ur- bano, and Anja Volk (eds.), Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, pp. 596-603 603, 2019. URL http://archives wee.eees.ss...... ismir. r pdf
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pp. 5670-5679. PMLR, 2018.
Yizhe Zhu, Martin Renqiang Min, Asim Kaday, and Hans Peter Graf. S3vae: Self-supervised sequential vac for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6538-6547, 2020.
A APPENDIX
In A. present implementation details of architecture, training process, and datasets. In A.2
present extra experimental results on synthetic data, including data reconstruction, more evaluation disentanglement, and more analysis physical symmetry and representation augmentation. In A.3 we show how our model performs on more realistic and complicated problems. Finally, we show autoencoder version of our method in A.4
12

---

Under review as a conference paper at ICLR 2023
A.1 IMPLEMENTATION DETAILS
A.1.1 ARCHITECTURE DETAILS
Out models for both tasks share the following architecture. The encoder is composed of a 2D-CN3 with ReLU activation connected by two parallel linear layers, one for estimating the mean and the other for the log-variance. The prior model is vanilla RNN of one layer with 256 hidden units and one linear layer projection head. The decoder consists of several linear layers followed by a set of 2D transposed convolution layers mirroring the CNN in the encoder. We apply the Sigmoid function to its output to produce the final output. We use no batch normalisation or dropout layers.
Minor variations exist between the models for the two tasks. In the audio task, we use three con- volution layers in the encoder, with three linear and three 2D transposed convolution layers in the decoder. In the vision task, as the data is more complex, we use four convolution layers in the encoder, with four linear and four 2D transposed convolution layers in the decoder.
A.1.2 TRAINING DETAILS
For both tasks, we use the Adam optimiser with a learning rate gradually decreasing from 10-$ to 10- in about 150000 batch iterations. The training batch size is across all of our experiments. For all VAE-based models we present, including SPS (ours), ours w/o symmetry. and 3- (base- line), we set | (i.e., A3 in Equation (T) to 0.01, with X1 = and X₂ 2. All BCE and MSE loss functions are calculated in sum instead of mean. K = 4 for all SPS models except for those discussed in section 5 where we analyse the influence of different K.
The RNN predicts I:TT given the first embeddings #1m..c. We choose = 3 for the audio task and n for the vision task. We adopt scheduled sampling (Bengio et al.) 2015) during the training stage, where gradually reduce the guidance from teacher forcing. After around 50000 batch iterations, the RNN relics solely on the given 4n+1:T,c and its predictions.
For random pooling in the training stage, one style vector is randomly selected from all time steps (i.e., 15 for the music task and 20 for the vision task) of the sequence to represent 2g. In the testing stage, only the first 5 (vision task) or 3 (music task) frames are given, and 2, will be selected from them.
Audio-specific setups. We run STFT (with sample rate 16000/s, window length 1024, hop length- = 512, and no padding) over each audio to obtain a spectrogram. We further convert its energy values into logarithmic scale and normalise them the range [0, 1). We then slice the spectrogram into fifteen segments, each containing one note. The CNN encoder, in each timestep, takes one segment as input, In practice, we sample ~~ 1[[ 1]). We set the dimension of ze and the dimension of 2, to 2.
In practice, we randomly choose from S G1 (R?,+) and S2 G2 = SO(2) to augment the representation. Both of them apply to two dimensions of ze. Under the constraints of symmetry these two dimensions represent the horizontal x-z plane. Similar to the audio setup, we sample S ~ ([-[1, 1J2). We set the dimension of ze to 3 and the dimension of =, to 2.
Vision-specific setups.
A.1.3 DATASET DETAILS
Music problem For the synthesized music scale dataset, each clips each containing 15 notes in major scales with the first half ascending and the second half descending. The tone of each note has the same volume and duration. The interval between every two notes equal. We vary the starting pitch such that every MIDI pitch in the range C2 C7 is present in the dataset. For each note sequence, we synthesise it using 53 different instruments, yielding 2376 audio clips. Specifically, two soundfonts are used to render those audio clips respectively: FluidR3 GM (Wen 2013) for the train set and General User GS v1.471 (Chris) 2017) for the test set. The pitch ranges for different instruments vary, so we limit each instrument to its common pitch range (See Table 15
Vision problem The simulated ball is affected by gravity and bouncing force (clastic force). A fixed camera records 20-frame video of each 4-second simulation to obtain one trajectory (see Figure
13

---

Under review as a conference paper at ICLR 2023
The ball's size, gravity, and proportion of energy loss per bounce constant across all trajectories.
Figure 9: Two example trajectories from the bouncing ball dataset.
A.2
EXTRA RESULTS
A.2.1 MORE INSPECTIONS ON LATENT FACTORS
To supplement results presented section Figure 10 shows a more quantitative analysis, using R as the metric to evaluate the linearity of the pitch against 2pitch mapping. Although SPICE produces rather linear mappings in Figure suffers from octave errors towards extreme pitches, hurting its R2 performance.
(Ours)
B-VAE (Baseline)
SPICE (Raseline)
Embedding
11
0.5
Synthesis
1.0-
n1
0.0
Clarinet
Ekectie Piant
Flate
Chiter
Saxophone
Trampel
Violin
Figure 10: We use R evaluate mapping linearity. larger R² indicates more interpretable latent space. Results are evaluated on the test set.
Figure evvalaaees the learned colour factor our model. Each pixel shows the colour of the ball synthesised by the decoder using different 2 coordinates. The ball colour is detected using naive saturation maxima. the central subplot, the location factor 21:3 stays at zeros while the colour factor 24:5 is controlled by the subplot's x. axes. As shown in the central subplot, our model (a) learns natural 2D colour space. The surrounding subplots keep the colour factor 24:5 unchanged, and the location factor 21.3 is controlled by the subplot's ar axes. A black cross marks the point where the entire 21:5 is equal to the corresponding black cross in the central subplot. As is shown by the surrounding subplots, varying the location factor does not affect the colour produced by our model (a), so the disentanglement is successful. The luminosity changes because the scene is lit by point light source, making the ball location affect the surface shadow. On the other hand, 8 3-VAE (b) learns an uninterpretable colour factor.
A.2.2 MORE ON CONTENT-STYIE DISENTANGLEMENT
To supplement sections 4.1.3 and 4.2.3 we further quantify the results the form of augmentation-
based queries following Yang al. 2019), regarding the intended split in as ground truth and the
14

---

Under review as a conference paper at ICLR 2023
SPS (Ours)
(b)
3-VE (Baseline)
R1
X
21 84
R1
X X
Figure 11: The colour map of the synthesised ball experiment through latent space traversal. Each pixel represents the detected colour from one synthesised image of the ball. Each subplot varies two dimensions of z, showing how the synthesised colour responds the controlled z.
dimensions with the largest variances from factor-wise augmentation after normalisation as predic- tions. For example, under timbre augmentation under given pitch for our model, if 2 and 23 are the dimensions of that produce the largest variances after normalisation, we count one false posi- tive (21), ) one false negative (22), and one true positive (2z). The precision would be 0.67. Tables 4 and [5| show the precision scores of the three approaches against their corresponding random selec- tion for the two tasks. The results are in line with our observation in the previous sections, with our model more likely produce the largest changes in dimensions in 2 under content augmentation and that in z under style augmentation.
Table 4: Results on augmentation- based queries on the audio task. Precision, recall and are the same since the number of predicted and ground-truth positives are identical. Note that random precisions for different approaches can be different as 2e and 2, are split differently.
Methods
Timbre augmentation
Pitch augmentation
Precision
Random Precision
Precision
Random Precision
Ours w/o z. splitting
0.50
0.67
0,02
0.33
GMVAE (Baseline)
0.93
0.50
0,83
0.50
TS-DSAE (Baseline)
0.81
0.50
0.68
0.50
SPS (Ours)
0.98
0.67
0.82
0.33
Table 5: Results augmentation--asee queries the visual task. Since the ablation model does not differently constrain 22 (corresponding to the y-axis) than 2,, we consider 2, and 2, differently for the two approaches. Under colour augmentation, consider 22 be a part of 2, for the ablation model and part of =, for the complete model. Under location augmentation, we consider to be a part of z, for both models.
Methods
Colour augmentation
Location augmentation
Precision f Random Precision
Precision f
Random Precision
Ours, w/o 2, splitting
0.64
0.60
0.36
0.40
SPS (Ours)
0.99
0.40
0.88
0.40
15

---

Under review as a conference paper at ICLR 2023
A.2.3 RECONSTRUCTION AND PRIOR PREDICTION
We investigate the video reconstruction and prediction capacities of our model and show that they are not harmed by adding symmetry constraints. We compare our model, our model ablating sym- metry constraints, and a 3-VAE trained solely for only image reconstruction. Table and Table respectively show the results of audio and vision tasks. For each task, we report per-pixed BCE of the reconstructed sequences from the original input frames (Self-recon) and from the RNN predic- tions (Pred-recon). We also include ppiirr the MSE loss on the RNN-predicted section 3.2
as defined in The results show that our model suffers little decrease in reconstruction and prediction performance while surpassing the ablation model in terms of Spiior.
Table 6: Reconstruction and prediction results of models in section 4.1.2 on the audio task.
Self-recon (BCE/Pixel) +
Pred-recon
Cprior (MSE/2)
Methods
(BCE/Pixel) 4
Ours w/o Symmetry
0.0360
0.0363
0.0486
3-VAE (Baseline)
0.0359
N/A
N/A
SPS (Ours)
0.0356
0.0359
0.0418
Table 7: Reconstruction and prediction results of models in section 4.2.2 on the video task.
Self-recon (BCE/Pixel) 4 (BCE/Pixel) 4
Pred-recon
Sprior (MSE/2) +
Methods
Ours w/o Symmetry
0.6456
0.6464
0.132
3- VAV (Baseline)
0.6455
N/A
N/A
SPS (Ours) 0.6457
0.6464
0.0957
A.2.4 MORE ANALYSIS ON REPRESENTATION AUGMENTATION
To supplement the results section 5 Figure 12 shows how representation augmentation jointly affect the linearity of the learned latent space and the reconstruction accuracy of é. Similar to Fig- 7 the experiment is conducted using different Ks, each with 10 trails. We see that representation augmentation does not sacrifice reconstruction but shrinks the linear projection loss in a consistent manner.
256 samples
512 samples
1024 samples
2048 samples
x10
K
1.208
Loss on x
1.206
1.201
Linear proj. Jass
Linear proj. losss
Lineur proj. loss
Linear proj. loss
Figure 12: The effect of representation augmentation on linear projection and the reconstruction x.
Additionally, we test SPS with deliberately incorrect group assumptions. The motivation is as fol- lows. In real applications, researchers may incorrectly specify the symmetry constraint when the
16

---

Under review as a conference paper at ICLR 2023
data complex the symmetry is not known a priori. SPS is more useful if it works with various groups assumptions close to the truth. In the following experiment, we are surprised to find that SPS still learns interpretable representations under five out of five alternate group assumptions that conceive by perturbing the correct group assumption.
Figure 13 shows results with the vision task (on the bouncing ball dataset). The ær tick labels show the augmentation method. Its syntax is introduced in section 3.3 but just for an example here, "(R², SO(2)" denotes augmenting representations by ID translations and 2D rotations. The axis of the plot is still linear projection loss (as discussed in section 4.2.2) that evaluates the interpretability of the learned representation. As is shown by the boxplot, five out of five perturbed group assumptions yield better results than the "w/o Symmetry" baseline. Particularly, (R", +) x SO(2) and (IR2, symmetry assumptions are "less incorrect" than others, and that SPS can achieve good results under multitude of group assumptions.
50(3) learn significantly more linear representations, showing that some
Linear proj. loss
0.6
0.4
0.2
0.0
w/o
(63
80(2)
(R)
(R)) ()
(R +)
(R3
Symmetry
S0(2) 80(2)
80(3)
S0(2)
(Ours)
Figure 13: Evaluation on various group assumptions. The y axis is linear projection loss between the learned location factor and the true coordinates, lower value means better interpretability of representations. The leftmost box shows the baseline without symmetry constraint. The next five boxes show five deliberately incorrect group assumptions. The rightmost box shows the correct group assumption.
A.3 MORE COMPLICATED TASKS
The main part of this paper focuses on simple, straight-forward experiments. Still, supplement our findings by reporting our current implementation' 's performance on more complicated tasks in- volving natural melody and real-world video data.
A.3.1 LEARNING INTERPRETABLE PITCH FACTORS FROM NATURAL MELODIES
We report the performance of SPS on learning interpretable pitch factors from monophonic melodies under more realistic setup. We utilize the melodies from the Nottingham Dataset (Foxley) 201 collection of 1200 American and British folk songs. For simplicity, we quantise the MIDI melodies by eighth notes, replace rests with sustains and break down sustains into individual notes. We synthesise each non-overlapping 4-bar segment with the accordion soundfonts in FluidR3 GM (Wen 2013), resulting in around 5000 audio clips, each of steps.
This task is more realistic than the audio task described in 4.1 since we use a large set of natural melodies instead of one specified melody line. The task is also more challenging as the prior model has to predict long and more complex melodies. To account for this challenge, we use a GRU (Cho) et al. 2014] with layers of 512 hidden units as the prior model. We perform early-stopping after around 9000 iterations based on spectrogram reconstruction loss the training set. The model and training setup is otherwise the same as in 4.1
We evaluate approach notes synthesised with all instruments in GeneralUser GS v1.471 (Chris 2017) in the MIDI pitch range of C4 to C6, where most of the melodies in Foxley (011 take place. Note that this is challenging zero-shot scenario since the model is trained on only one instrument. We compare our model, our model ablating the symmetry loss and a 3 3-VAE baseline. We visualise the embedded zpitch and synthesised pitches for different instruments in
Following 4.1.2
17

---

Under review as a conference paper at ICLR 2023
Even when tested on unseen timbres, our model can learn linear and interpretable pitch factors and demonstrates better embedding and synthesis performance compared with the ablation model, which outperforms the 3-VAE baseline.
Figure 14
Following A.2.1 R results shown in Figure 15 and Table 8
SPS (Ours)
Ours w/o Symmetry
VAE (Baseline)
Accordion
Guitaz
Embedding
Saxophone
Violin
Epttch
Clariuel
E. Piano
Flute
Trumpet
04
C6
C4
06
C4
C6
Pitch
Pitch
Pitch
C6
CG
C6
Detected Pitch
Synthesis
C4
C4
C4
Spitch
Sprsch
Spitch
Figure 14:
visualisation of the mapping between the embedded ID content factor and the true pitch for the model trained on Nottingham dataset.
SPS (Ours)
BB
Ours w/l Symmetry
XXX
3-VA (Baoeline)
Embedding
0.0
Synthesis
1.0-
0.5-
Accordon
Accordoot
Clurioes
Eestric Piano
Fatte
Quitar
Sanephooe
Violn
Figure 15: R² for select instruments in the test set. A larger R² indicates more linear and inter- pretable latent space.
A.3.2 LEARNING AN INTEPRETABLE LOCATION FACTOR FROM KITTI-MASKS
In this task, we evaluate our method's capability on a real-world dataset, KITTI-Masks (Klindt et al. 2021). The dataset provides three labels for each image: X and Y for the mask's 2D coordinate, and AR for the pixel-wise area of the mask. Based on the provided labels, we use simple geometrical relation to estimate the person-to-camers distance d, computed as d = 1/ (A(()) where a is constant describing the typical camera's 's Field of View (FoV).
We use a 3-dimensional latent code for all models. For SPS, all 3 dimensions are content factors &, and style factor z, is used. We apply group assumption (R3, +) augment representations with
18

---

Under review as a conference paper at ICLR 2023
Table 8: R aggregated across all instruments in the test set. A larger R
indicates a more inter-
pretable latent space.
Method
Embedding R
Synthesis R
SPS (Ours)
0.89
0.47
Ours w/o Symmetry
0.83
0.25
B BAV (Baseline)
0.19
0.29
K ===1 1. To measure the interpretability, fit linear regression from to the ground truth labels and calculate MSEs the same way as in section 4.2.2 The results are shown in Table 9 Linear proj. MSE measures the errors of linear regression from 7e the original dataset labels. Linear proj. MSE measures the errors of linear regression from 2 to the person's 's 3-D location, estimated from the labels.
As is shown in Table MSE is smaller than MSE for SPS, indicating that SPS learns more fun- damental factors (person's location) rather than superficial features (pixel-wise location and area). For the baseline methods, MSE 2 is almost equal to MSE 1, and both of them are higher than those of SPS. In summary, our experiment shows that SPS learns more interpretable representations than the baseline (as well as the ablation method, "Ours w/o Symmetry") on KITTI-Masks dataset.
Table 9: Results of KITTI-Masks task, averaging on 30 random initialisations for each method.
Self-recon (BCE/Pixel) |
Pred-recon
Linear proj. MSE
Linear proj-
Methods
(BCE/Pixel) L
MSE 2 4
Ours w/o Symmetry
0.030++0.001
0.0010
0.243++0.08 0.235
3- BAE (Baseline)
0.028-+0.001
N/A
0.403 194
0.399
SPS (Ours)
0.030110.001
0.084:100006
0.215:10.067
0.2031
Figure [ojillustrates that the factors learned by SPS are more linear than those learned by other meth- ods in the human location attribute. We choose all sequences with length 2 12 from KITTI-Mashe as our dataset. Specifically, we use 1058 sequences for training and 320 sequences for evaluation. In the inference stage, only the first 4 frames are given. All three methods are trained 30 times with different random initialisations. Table shows the average over the 30 random initialisations, evaluated on the same test set.
(a) SPS (Ours)
(b) Ours w/o Symmetry
(c) B-VAE (Baseline)
25
2.0
0.0
Figure 16: Latent space traversal on different models. Row shows the generated images when changing and keeping Zz4 = 0. The range of 21 from -2 to 2 corresponds to the human location from near-right far-left, 22 from near-left to far-right, and 23 from near far. We can see that other methods produce more non-linear trajectories, for example in (c), the human location hardly changes when 21 O, but changes dramatically when 21 >
19

---

Under review as a conference paper at ICLR 2023
A.4 SPS WITH AUTOENCODER (AE)
We also test our method on AE, i.e., a model with almost the same architecture but the latent rep- resentation variational. Although AE usually suffers from its unregularised latent distribution and cannot learn meaningful embeddings, we find that once regularised with physical symmetry AE out learn interpretable latent representations. This section presents our preliminary findings.
Table 10 supplements Table 2 by adding AE to the comparison. The latent space learned by AE is less linear than that learned by VAE but still much better than the ablation and baseline methods.
Table 10: Linear fits between the true location and the learned location factor, comparing AE with VAE.
Method
x axis MSE 4
y axis MSE +
axis MSE
MSE
SPS-AE
0.17
0.23
0.24
0.21
SPS-VAB
0.11
0.06
0.09
0.09
We further evaluate AE's disentanglement performance following the methods in sections 4.1.3
4.2.3 and A.2.2 Figure 17 Table 11 and Table 12 respectively supplement Figure Table 1 and Table4 Figure 18 Table 13 and Table supplement Figure 6 Table 3 and Table5 All results are evaluated on test sets. The AE variant of our model achieves comparable performance to the VAE both pitch-timbre and space-colour disentanglement, even slightly surpassing VAE in several metrics. Our results show that symmetrryrregulariittion empowers AE to learn well-disentangled representations.
|Az.||a
|A l2
SPS-VAE
SPS-AE
Clarinos
E: Piano
Plute
Guitar
Piano
Suxophoue
Trumpet
Vielin
Clarinet
Plute
Guitar
Plano
Soxophone
Trumppp
Violiu
Piano
Figure 17: Comparisons for Aze and Az, for different instruments against accordion, with pitch kept constant at MIDI pitch D3.
Table 11: Mean ratios of že and changes under timbre augmentation across all possible instrument combinations under different constant pitches. We compare the AE and VAE variants of our model.
Methods
SPS-AE
0.48
SPS-VAL
0.49
Figure is plotted in the same way as Figure g showing how the true location- space map- ping becomes gradually more disentangled and linear training progresses. AEs, once symmetry- augmented, are able to learn a location factor of similar qualities to those VAEs within the same number of epochs.
20

---

Under review as a conference paper at ICLR 2023
Table 12: Results augmentation- queries the audio task.
Methods
Timbre augmentation
Pitch augmentation
Precision t
Random Precision
Precision f
Random Precision
SPS-AE SPS- VAE
0.99
0.67
0.80
0.33
0.98
0.67
0.82
0.33
|As.1la
| la
SPS- VAE
SPS-AE
2
2
Blue
Green
Pink
Yellow
Blue
Cyen
Green
Pink
Red
Yollow
Figure 18:
Comparisons for Azo and Az, for different colours against white, with location kept
constant at (0, 5)..
Table 13:
Mean ratios of ze and changes under colour augmentation across sampled colour com- binations. Locations are kept unchanged.
Methods
/ Az,
SPS-AE
0.48
SPS-VAE
0.54
Table 14: Results augmentation-bassd queries the visual task.
Methods
Colour augmentation
Location augmentation
Precision t
Random Precision
Precision t
Random Precision
SPS-AE 3PS-VAE
0.98
0.40
0.85
0.40
0.99
0.40
0.88
0.40
SPS-AE, Representation A Augmented by 4x
2
SPS- VAH, Representation Augmented by 4x
24
22
-2
2
-2
22
2
22
2
2
22
2
-2
-2
2
-2
-2
2
-2
2
22
-2
2
-2
Epoch
331
1000
1381
1069
2000
2331
2069
3000
3331
3009
1000
1331
1009
5000
Figure 19: Comparing VAE and AE in terms of latent space regularisation during training.
21

---

Under review as a conference paper at ICLR 2023
Table 15: Pitch range (in MIDI note) for each instrument in our dataset.
MIDI Note
MIDI Note
Instrument
(from)
(to)
Accordion
58
96
Acoustic Bass
48
96
Banjo
36
96
Baritone Saxophone
72
Bassoon
36
84
Celesta
36
96
Church Bells
36
96
Clarinet
84
Clavichord
36
84
Dulcimer
36
84
Electric Bass
40
84
Electric Guitar
36
96
Electric Organ
36
96
Electric Piano
36
96
English Horn
36
85
Flute
48
96
Fretless Bass
36
84
Glockenspiel
36
96
Guitar
36
96
Harmonica
36
96
Harp
36
96
Harpsichord
36
96
Horn
36
96
Kalimba
36
96
Koto
36
96
Mandolin
36
96
Marimba
36
96
Oboe
36
96
Ocarina
36
96
Organ
36
96
Pan Flute
36
96
Piano
36
96
Piccolo
48
96
Recorder
36
96
Reed Organ
36
96
Sampler
36
96
Saxophone
36
84
Shakuhachi
36
96
Shamisen
36
96
Shehnai
36
96
Sitar
36
96
Soprano Saxophone
36
96
Steel Drum
36
96
Timpani
36
96
Trombone
36
96
Trumpet
36
96
Tuba
36
72
Vibraphone
36
96
Viola
36
96
Violin
36
96
Violoncello
36
96
Whistle
96
Xylophone
36
96
22


Reviews:
[{"review_id": "kc6nQXbLOPE", "reviewer": "Reviewer_rnjU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes an auto-encoder setup which aims to disentangle content and style using multiple inductive biases. (1) The style given by selecting random times-slices of the encoded time series, (2) the encoded time series need to be predictable 1-step-ahead using a recurrent network R and (3) the encoded time series need to be equivariant to random translation or rotations S. The setup is tested on two synthetic datasets: a synthesized audio dataset with ascending and descending note sequences, and an short video clips of a bouncing ball. On these datasets, pitch or ball position seem to be well separated from the timber and the instrument texture.", "summary_of_the_review": "The idea is simple and well described, but it is not clear whether this method would be capable to disentangle content and style on a real dataset because (1) it is only tested on simple synthetic datasets and (2) some simplifying 1D-latent assumptions are necessary for the network architecture to reconstruct the input faithfully.\n\nI believe I will increase my score if my comments are addressed appropriately and I become sure that the experiments and the Figures are flawless and the paper is transparent about the computed performance metrics (is that train or test? how many instruments? normalization? why choosing beta-VAE as a baseline and then GMVAE etc...)", "strength_and_weaknesses": "The paper is enjoyable to read and the idea is simple and well described.\n\n1) I think a more precise way to describe the inductive bias of the network architecture is that it \"separates some latent equivariant spatio-temporal features from time-invariant features\". I think it would be more didactic to spell this out in the intro and intuition section because currently the description remains on the high level description of \"physical symmetries\" or \"pitch vs. timbre\" disentanglement which I find less precise and specific to this work.\n\n2) Disentanglement of content versus style is an important topic, for instance to separate pitch, timbre and instrumental features. I am not convinced however that this technique has a strong reason to extract \"only\" the pitch with arbitrary audio because there are many other 1D features in music which can be described with a latent equivariant temporal series: loudness, note frequency, RMS ratio of high frequency vs. low frequency etc.. All these features are likely to be independent from the pitch in a real dataset and may well be what is extracting in to the latent time series z after convergence. If there is no way to prove this wrong, it might be good to report that in the discussion.\n\n3) For the reason above, I expect that the inductive bias of the model would not be expressive enough to describe the full dataset. So I would expect a substantial loss of information in the latent representation of the auto-encoder. In fact this might even be true on the simple synthetic datasets that are considered because there are no metric report the \"quality\" of the reconstruction against the beta-VAE for instance. Maybe the beta-VAE focuses on other important features which are more crucial for the reconstruction of the audio than the pitch? Or for the same argument as before, the re-constructed audio is likely to have a wrong timber, loundness, etc.. if the pitch is the only time-variable latent that is captured. Reporting a metric of audio reconstruction quality would be a good start.\n\n4) One possible flaw: the number of samples in the training sets look ridiculously small. I suspect that the model might not generalize well to the test/validation set in this setting. It is therefore crucial to state explicitly in the captions of ALL figures and tables if performance metric (or plotted data) is shown for samples from the training or testing sets.\n\n5) It is not very standard to put the related work in discussion rather than introduction. It makes the whole idea of group symmetry sound a lot model novel that it is at first sight. I would encourage the authors to put at least a few references about disentangling AEs and group symmetry in the intro for this purpose. Also i am wondering but I am not sure, wouldn't it be fair to cite some papers from the Max welling group who have used extensively symmetry groups to generate equivariant representations? Here are two suggestions:\n\nhttp://proceedings.mlr.press/v121/ilse20a.html\n\nhttp://proceedings.mlr.press/v48/cohenc16.html\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K30yKu3oT9s", "reviewer": "Reviewer_AY1K", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to train sequence models that encourages disentangling physical/dynamic features (\"content\") from constant features (\"style\"). The method partitions the latent features into two subsets and enforces certain symmetries by applying appropriate random transformations and penalizing the latent transition model (an RNN) for deviations from equivariance. Experiments on synthetic data show that the content features take on straightforward interpretable meanings, and that style changes in the input do not affect the content features.", "summary_of_the_review": "This is a solid contribution. My main criticism is the applicability of the method beyond very simple settings.", "strength_and_weaknesses": "Strengths:\n  - The approach is simple and sound.\n  - It is easy to understand what is being done.\n  - The experiments are convincing, if contrived (more below).\nWeaknesses:\n  - There is no discussion of the random pooling on the style. This random pooling enforces its own symmetry -- it encourages the style representation to be constant over time. I believe that without the random pooling, the method would not work; the model would choose to convey all its information through the unconstrained style channel and not bother with the content channel.\n  - Related to the previous point, the effectiveness of the method hinges on being able to partition the latent space into subspaces and coming up with appropriate symmetries for all subspaces. The experiments are performed on synthetic data that is contrived to have this exact physical-content-plus-constant-style breakdown, and it is unclear what one would do in real-world settings where this breakdown does not make sense.\n  - The experiments that study the impact of style changes in the input use norms to measure the extent of change in the latents. Norms aren't really meaningful here; a small delta norm ||dx|| may correspond to a large change if ||x|| << ||dx||, and a large delta norm ||dx|| may correspond to a small change if ||x|| >> ||dx||. Consider using relative norms ||dx||/||x|| instead.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GLLdTPGfaJE", "reviewer": "Reviewer_9KCD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes novel ways to enforce an inductive bias into a temporal auto-encoder model, by:\n1) enforcing that the R part of the model should be able to operate on both the original version of the time-varying embeddings and also on a transformed version.\n2) separating out the per-session and per-frame parts of the embeddings.\n\nThe motivation of enforcing 1. is to allow for an inductive bias that does not require domain expertise to design.  However, the proposal requires domain expertise to choose the S transformation, which goes against the original motivation.\n", "summary_of_the_review": "The proposal is motivated as being able to avoid domain expertise, but in the end domain expertise is needed to choose S.\nThe design choices are not well motivated in the text.\n\nIthenticate similarity is 3%, which is good.\n", "strength_and_weaknesses": "Strengths:\n- It is not trivial to experimentally assess whether the proposal is doing what the authors claim that it should be doing.  The authors go through a lot of effort to run many different types of innovative tests to assess this behaviour.\n\nWeaknesses relating to unclear description of the proposed approach:\n- How do you split z into z_s and z_c?\n- Why does transforming z through S yield a fake sequence?  What does \"fake sequence\" mean in this context?\n- The model described in figure 2 does not have any K.  Yet, section 3.3 mentions K different transformations.  Are the descriptions consistent?\n- In the beta-VAE baseline in figure 3, how is the pitch estimated from the multi-dimensional embeddings?\n- What is a \"bouncing force\"?\n\nWeaknesses relating to lack of motivation for why design choices were made:\n- What is the motivation for using random pooling in P, as opposed to mean pooling or self-attention?\n- What is the motivation for having each of the three branches in the model?\n- In the introduction, the proposal is motivated as being better than previous works, because there is no need for domain expertise.  However, S needs to be chosen with domain expertise.  Is the original motivation still satisfied?\n- What is the motivation and mathematical definition of each term in the training criterion?  \"BCE\" is not defined in the paper.  l_2 is not defined.  Why does the KL divergence measure a distance between z_i and a standard Gaussian?\n\nWeaknesses relating to ambiguity in the language used:\n- What does \"group assumption\" mean?\n\nWeaknesses relating to questionable design choices:\n- It is claimed that the proposed inductive bias separates out pitch and timbre, or location and colour.  How do you know that these are the only factors involved?  What about other factors, such as volume, speed, starting location, etc.?\n- In section 4.1.3, how do you know that the difference in magnitude of delta z_c and delta z_s is not because z_c and z_s learn to have different dynamic ranges?  Perhaps delta z_c and delta z_s should first be normalised by the dynamic ranges of z_c and z_s.\n- In section 4.2.2, why do the movement directions conveniently align with the standard basis of z?  Why do they not align with an arbitrary rotation of the standard basis of z?  Maybe the movement directions are also aligned with an arbitrary rotation of the z basis for beta-VAE?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3Rmr7H46yN_", "reviewer": "Reviewer_5zXw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a self-supervised learning method that learns disentangled and interpretable representations based on physical symmetry from time-series data. The experiments with monotonic instrument sounds and monocular videos of bouncing ball validate the method by showing disentanglement in pitch-timbre and local-color, respectively. In addition, they suggested the representation augmentation technique that lead to lower linear projection loss and improves the disentanglement. \n\n \n\n", "summary_of_the_review": "This paper is very interesting to read and the intuition from the fields of physics is implemented well as a model that learns disentangled and interpretable representations without any labels. Also, it suggests the novel representation augmentation technique for more effective model training. The experiment and analysis clearly show that  the model successively achieves the goal. \n\nHowever, my overall impression is that this paper is in the stage of proof-of-concept. The example data are quite simple: single harmonic pitch and single ball object. The experiment focuses on decoupling the latent space into locational features (pitch in spectrogram and ball position) and the rest. It is not trivial to apply the proposed model to more complex data where the content latent features change over time in a more complicated manner (e.g., polyphonic music). The authors should clearly envision the potential of the proposed model that be applied to the complex data. \n\n", "strength_and_weaknesses": "Strengths\n- The application of physical symmetry to unsupervised representation learning is very intriguing.\n- Experimental results clearly validates that the model learns the disentangled and interpretable representations as targeted.\n- The representation augmentation idea sounds very novel. The experiment shows the effectiveness well. \n\n\nWeaknesses\n- The dataset used to validate the proposed learning model are too simple: monotonic musical instrument and monocular videos of bouncing ball. It is not clear how the model can be extended to more complicated data, for example, polyphonic and multi-instrument music and real-wold videos.\n- If the model is applied to a very long time-series data, RNN would have a limitation in learning the long-term dependency. If the Transformer model is used instead, it can be plugged in the proposed model?  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject