Symmetric Space Learning for Combinatorial Generalization

Authors: Jaehyoung Jeong, Hee-Jun Jung, Kangil Kim
Primary area: generative models
Keywords: Generative Model, Generalization, Combinatorial Generalization, Machine Learning, Manifold Learning, Representation Learning

Abstract:
Symmetries on representations within generative models have shown essential roles in predicting unobserved combinations of semantic changes, known as combinatorial generalization tasks. However, these efforts have primarily focused on learning symmetries from only training data, and thus, the extension of trained symmetries to unseen samples remains uncontrolled. A potential approach for generalizing the symmetries is leveraging geometric information on manifolds that contain functional semantic structures for unseen data, but it still falls short of supporting symmetry learning. In this paper, we address this $\textit{symmetry generalization}$ by forcing $\textit{symmetric space}$ on latent space for utilizing semantic structures from symmetry and manifold perspectives. We clarify an equivariance-based constraint that restricts symmetry generalization, and prove that: 1) enforcing the homogeneous space property of symmetric space onto the data manifold eliminates this constraint, 2) a homogeneous latent manifold induces the data manifold through diffeomorphic data-to-latent mapping, and 3) the isometry property of symmetric space extends neighbor symmetries of a point to another within the space. For practical implementation, we propose a method to align sampled points from symmetric space with their explicitly trained geodesic. We verify the method in a detailed analysis on a toy dataset and enhance combinatorial generalization on common benchmarks. This work represents the first effective effort to align symmetries with manifolds for combinatorial generalization.

Decision: reject
Reviewer scores: 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729445469461

thread_type:
official_review

<thread_object>
creator: reviewer_cbnk

<content>
summary:
This paper studies the symmetry generalization problem in the latent space of a dataset. Specifically, the authors assume the data manifold and its latent space are homogeneous with some group actions, then the unseen samples can be generalized by some group actions on the existing samples as the group is assumed to act transitively.

strengths:
The authors propose a novel approach through learning the symmetry in the latent space to solve the combinatorial generalization problem in the data space.

weaknesses:
1. The paper is not carefully written with several grammatical errors. For example, the first sentence of Proposition 2; between line 172-173, "However, encoding data onto a latent manifold enables data to be handled in lower dimensional and easy-to-compute data."

2. Condition 3 is the key assumption of this paper, but it is too strong without any justification.

3. I'm confused by the last sentence between line 174-175, is $\mathcal{N}\subset\mathcal{Z}$ or vice versa?

4. The math part is presented in a redundant way. I think the authors can make it more clear by clarifying all the assumptions at once. Also, the propositions and theorems are trivial, and they are essentially the mathematical formulations of the assumptions.

5. Condition 4 is also a very strong assumption, and it requires $\mathcal{M}$ to have the same dimension as $\mathcal{N}$, so why is $\phi$ an encoder?

questions:
See Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730244395703

thread_type:
official_review

<thread_object>
creator: reviewer_w48h

<content>
summary:
This paper addresses the problem of combinatorial generalization onto unseen data by enforcing the latent space to be a symmetric space. To achieve this, the paper proposes several loss terms to enforce the geodesic symmetry in the learned latent. The method is tested on standard datasets for combinatorial generalization, including dSprites and 3D Shapes.

strengths:
The paper discusses an interesting topic, combinatorial generalization using symmetry-based representation learning.

weaknesses:
The paper is poorly written. It contains many grammatical errors, confusing phrases, and informal statements, and it lacks coherence overall. As a result, the goal and the contribution of this paper are totally unclear to me, which is my main concern before all others. As far as I can understand, the authors are trying to address the problem in combinatorial generalization that symmetry transformation may take data points outside the training domain. This problem is indeed valid, but I fail to see how the theories in Section 3 or the method in Section 4 can solve it. In particular,
* It seems that the idea of this paper is to learn a latent space which is a homogeneous space by enforcing the model to align with the geodesic symmetry (please correct me if this is wrong). But there is not enough explanation as to why this would work. I notice Thm 3 does mention something related, but it is stated informally and vaguely (e.g. What does it mean by "make symmetric space **enough**"? ) and the proof is missing.
* The method involves approximating the geodesic. It simply parameterizes an arbitrary path in the ambient and penalizes its length in the interval $t \in [0, 1]$ and the reversal symmetry error at a single point $t=-1$. There is no justification for why (1) this would result in a path that stays on the latent manifold (moreover, the latent manifold itself has not been identified anywhere in the discussion), (2) the learned path function would generalize to $t<0$, and (3) the geodesic symmetry would hold for all $t$ instead of only at $t=1$.
* The role and properties of the anchor $a$ are also not adequately explained. It is unclear whether any constraint needs to be enforced for this learnable anchor to make sure its properties match those in the definition of symmetric space.
* I am also confused by the fact that, generally in symmetry-based representation learning, the symmetry transformations represent the factors of variations, and depending on specific data generation procedures there should be different symmetry groups, while this paper seems to only consider the geodesic symmetry as defined in Defn 3.

Some other less major concerns, mainly about presentation and organization of the paper:
* Definitions in Section 2: instead of making a separate paragraph title (e.g. symmetric space), you can specify the terminologies right after the definition header (e.g. Definition 3 (symmetric space)).
* Definition 3: is there a confusion between $o$ and $a$? Should it be $T_a \mathcal M$? Also, what does it mean by "$U$ can be extended to the entire manifold"?
* Section 3 can be made more concise. The proofs are too verbose. Statements like Prop 2 and Thm 1 just directly follow from definitions.
* Thm 1: Let $X, X'$ be partitions of a manifold $\mathcal M$. This statement is misleading. Partition can mean different things according to context. If the authors mean these two sets are disjoint and their union is the entire manifold, a better way to phrase it may be ``$\\{X, X'\\}$ is a partition of $\mathcal M$''.
* The training objective, Eq (9), involves a base loss from another paper called MAGANet. This name came out of nowhere for the first time in L315. The authors should explain this in more detail, or at least reference the original paper here. Also, the authors should discuss whether the proposed method can only be applied to MAGANet, which would be a potential limitation.

questions:
* What is $\epsilon$ in Eq (2)? Is it a random noise? What is the reason for adding a noise component to the path $\gamma$?
* Can the proposed method be applied to models other than MAGANet?
* Why do the results in Table 1 have such large differences? Does a BCE of 3000+ indicate total failure? Also, for the 3D Shapes dataset, $\beta$-VAE ($\beta=2$) seems to perform best in all cases. Why is your method highlighted then?
* In Fig 1 caption: Why is the volume form related to the likelihood?
* Can you elaborate on why you need to exclude three specific combinations of factors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714047207

thread_type:
official_review

<thread_object>
creator: reviewer_dkpg

<content>
summary:
This paper proposes a novel method for symmetry generalization -extending trained symmetries on observed data to unseen data for combinatorial generalization. In this approach, the latent space manifold is enforced to be a symmetric space, and homogeneous conditions on the data manifold as well as the latent manifold are present. The sampling method proposed here explicitly learns geodesic symmetry, which helps reduce the computational costs for sampling and induces symmetric space for latent manifolds. 
The paper presents empirical experiments through qualitative and quantitative analysis on datasets like MNIST, dSprites and 3D-Shapes.

strengths:
1. The paper presents a well-defined idea to build a non-rigid approach to building symmetries and uses this for combinatorial generalization. 

2. The proposed method builds a local structure in the latent manifold which can therefore be used for geodesic symmetries. I believe this approach can be useful for different tasks like latent disentanglement.

weaknesses:
1. The proposed method uses one object per data sample, be it MNIST, dSprites, or 3Dshapes, such that the geodesic symmetry approach works with any issues of entanglement. It would be interesting to show how this extends in multi-objects set-up.

2. The homogeneity conditions proposed for both data manifold and latent manifold, might not strictly hold in the case of 2 or more objects in a sample.

questions:
1. How does strict permutation invariance in latent generative models compare to this approach? 
2. Previous works like [1,2] are capable of representing invariant shapes as latent. How does this explicit and exact symmetry equivariant VAEs compare to this work?
3. Can this approach be extended to hierarchical latent models like NVAE? 



[1] Continuous Kendall Shape Space VAEs, Vadgama et al

[2] EqVAE:Equivariant Priors for Compressed Sensing with Unknown Orientation. Kuzina et al.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730771124101

thread_type:
official_review

<thread_object>
creator: reviewer_ycwt

<content>
summary:
The authors propose a method to enforce the latent space to be symmetric in order to generalize combinatorially to symmetries. By first learning the symmetric latent space from training data, the network can generalize to combinations of symmetries at inference. Specifically, they learn the geodesics of the latent manifold and extrapolate points on the geodesic by reflection. The authors provide theoretical results on how enforcing homogeneity and isometry on the latent space and using an equivariant map induces the data manifold to also be symmetric. Experiments on MorphoMNIST and dSprites show better results than baselines.

strengths:
- The authors clearly define a symmetric space and provide proofs on how properties of the symmetric space lend themselves to unseen combinations of symmetries
- The practical implementation seems relatively straightforward and is clearly described.
- The experiments show favorable results compared to MAGANet and BetaVAE.

weaknesses:
- The theoretical results seem somewhat obvious and have been shown before [1]. Furthermore, there seems to be a disconnect between the theory and the practical implementation. Could the authors clarify how enforcing the various loss terms ensures that the encoder is an equivariant diffeomorphism?
- The experiments were done on somewhat toy domains. In particular, I feel that the 3D-Shape dataset shows the objects in different colors relative to the background, making the task easier. What would happen if the 3D-Shapes dataset did not include hue?
- Baselines: MAGANet seems to be the only baseline that aims to learn the symmetries directly. It would have been nice to consider other methods that learn symmetries [2], [3]

References:
1. Cohen, T. S., Geiger, M., & Weiler, M. (2019). A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32.
2. Falorsi, L., De Haan, P., Davidson, T. R., De Cao, N., Weiler, M., Forré, P., & Cohen, T. S. (2018). Explorations in homeomorphic variational auto-encoding. arXiv preprint arXiv:1807.04689.
3. Miyato, T., Koyama, M., & Fukumizu, K. (2022). Unsupervised learning of equivariant structure from sequences. Advances in Neural Information Processing Systems, 35, 768-781.

questions:
- Figure 3: are the black lines the true geodesic? How were they computed?
- How do you find the anchor for geodesics?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work claims that symmetric latent spaces can address the problem of combinatorial generalization, and several loss terms are proposed which enforce latent symmetry. Improvements on dSprites and 3D Shapes datasets are claimed. 

While the presented approach on learning symmetric latents using geodesics is interesting, there are several issues with the paper such as correctness of Proposition 1 and 3 as pointed out by reviewer W48H. While the proposed algorithm and framework is interesting, the numerical results are done on toy domains (dSprites and 3D Shape) and larger scale evaluations would make the work more convincing. At this stage the paper does not meet the standards for acceptance at ICLR and encourage the authors to take the reviewers feedback into account for a resubmission.