Learning Group Actions on Latent Representations
Authors: Yinzhu Jin
Aman Shrivastava
Tom Fletcher
Venue: NeurIPS 2024 poster (2024)
Keywords: group action
representation learning
image rendering
Primary area: deep_learning_architectures
TL;DR: A model to learn group action on latent factors, e.g. manipulating objects on 2D images.

Abstract: In this work, we introduce a new approach to model group actions in autoencoders. Diverging from prior research in this domain, we propose to learn the group actions on the latent space rather than strictly on the data space. This adaptation enhances the versatility of our model, enabling it to learn a broader range of scenarios prevalent in the real world, where groups can act on latent factors. Our method allows a wide flexibility in the encoder and decoder architectures and does not require group-specific layers. In addition, we show that our model theoretically serves as a superset of methods that learn group actions on the data space. We test our approach on five image datasets with diverse groups acting on them and demonstrate superior performance to recently proposed methods for modeling group actions.

Learning Group Actions on Latent Representations
Yinzhu Jin
Aman Shrivastava
Department of Computer Science
Department Computer Science
University of Virginia yj yj3cz@virginia.er
University of Virginia
as3ek@virginia. edu
P. Thomas Fletcher
Department of Electrical and Computer Engineering Department of Computer Science University of Virginia ptf8vvvirgina.. edu
Abstract
In this work, we introduce new approach to model group actions in autoencoders. Diverging from prior research in this domain, we propose to learn the group actions on the latent space rather than strictly on the data space. This adaptation enhances the versatility of our model, enabling it to learn a broader range of scenarios prevalent in the real world, where groups can act on latent factors. Our method allows wide flexibility in the encoder and decoder architectures and does not require group-specific layers. In addition, we show that our model theoretically serves superset of methods that learn group actions on the data space. We test our approach on five image datasets with diverse groups acting on them and demonstrate superior performance to recently proposed methods for modeling group actions.
Introduction
Group actions natural mathematical representation of symmetries and geometric transformations of data. Recent work has demonstrated that explicitly modeling and learning such group actions in neural networks be beneficial for many tasks, such learning latent representations [15, 22, 23], generative models [8, 12, 34], and classifiers [1, 4]. While many existing works model group actions on the data space, to the best of our knowledge, nearly all prior works have overlooked group actions on latent factors.
However, there are certain scenarios where desire to model group actions on factors that not directly observed. Figure provides an illustrative example. The top row shows a rotating image of '7', i.e., orbit under the group action of (2( the image space. (Technically, because of image interpolation, this is only approximately a group action.) This can be effectively modeled using existing approaches designed to learn group actions the data space. However, if we introduce slight modification, adding a fixed block, then the digit will be partially occluded for some rotations. The occluded part of the digit is absent from these images but reappears when the digit continues rotate. This new scenario can no longer be modeled as group action on the data space. Instead, the group is now acting on the underlying factors. By encoding the representation of these latent factors, specifically the digit in this example, and learning the group action on it, we will be able to correctly model this scenario. While this is a synthesized example, does exemplify common phenomenon in real- real-world images, namely occlusion. Other scenarios in real-world data involve group actions on latent factors. For example, consider taking photographs of rotating 3D object. The rotation group will be acting on the 3D object geometry but not on the resulting 2D images.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure I: Top row: group action on the image; bottom row: a group action on the digit but not on the image
We propose to learn group actions in the latent space, without the necessity to conform group actions on the data space. Employing an autoencoder framework, we learn the latent data representations on which the group acts. Our method is not confined to any specific group or certain set of groups. It also does not depend on any group-specific layers for encoding and decoding the data, i.e., allows wide flexibility in the architecture of the autoencoder, as long as it is expressive enough. We present several examples of groups to explain how our model can be applied. Furthermore, we theoretically demonstrate that model is capable of learning group actions on the data space as well, positioning as an extension of general data space group action modeling methods.
In summary, there are four key elements to contributions:
We propose method to learn group actions in the latent space, which allows our model to perform group actions on the latent factors.
Our approach flexible and can be applied different groups without speup-specific architectures to obtain latent representations. This enables the model to benefit from any advanced deep learning frameworks.
We focus on rendering new data given the applied group action rather than merely learning the representation.
The proposed strategy can be seen extension of prior works modeling group actions in the data space. It can still be used to model group actions on the data space, which we show is special case of the more general setup.
Related works
There has been considerable research on incorporating group actions into data representation learning and to benefit downstream tasks. For instance, Dey et al. [8] introduces a GAN-type model with discriminator that equivariant to the p4 or p4m group, resulting in enhanced generative performance even with limited training samples. Moreover, instead of learning the group actions in the data space, Park al. [21] learn representations equivariant to the latent group actions with a contrastive loss. Wang et al. [30] propose to exploit extrinsic equivariance to model latent representation symmetries and demonstrate benefits in a downstream reinforcement learning task.
Furthermore, part of these works are interested in rendering new data by applying group actions. A significant portion of the studies concentrates on specific group that acts on the data space. To give an example, Hoogeboom et al. [12] put forth diffusion model that maintains equivariance 3D Euclidean transforms generation. In contrast, Yim et al. [34] propose generative model of protein backbones, modeled as a product of rigid body motions, using a manifold diffusion model [2]. These approaches rely on architectures tailored to be equivariant to specific group a set of groups Cohen and Welling [4] first introduced a group-equivariant convolutional neural network (CNN) featuring group-qquivariant convolutional layers. This architecture is designed for groups that represent discrete transformations, including translations, rotations, and reflections. This was followed by multiple works extending group equivariance to broader classes of groups and architectures [5, 23, 31]. In another vein, Satorras et al. [26] present an E(n) equivariant graph neural network, where each graph convolutional layer is inherently designed to be -egg-v-vvrvriant.
each denoising time step, specifically tailored for 3D molecule
In alignment with our proposed method, several other works strive to construct models with more flexibility on the types of groups involved. For example, Quessard et al. [22] parameterize group representation as a product of rotation matrices. The model assumes a finite number of group elements and learns representation for each. Training involves sequential data collected from series of

---

group elements acting on the underlying generative factor in known order. It is noteworthy that this model uses the product of groups itself as the latent representation. However, a key distinction is that it cannot effectively model group actions on sets containing more than one orbit, due to its direct mapping from the group to the data space. On the other hand, Winter et al. [32] propose learning latent representations consisting of group-invariant components and group elements for acting on the data space. While their framework is not specific to any particular group, the model requires group-specific architectures capable of generating group-invariant representations, e.g.. steerable CNNs [S]. In another approach. Hwang et al. [13] suggest predicting the group element that transforms one data point to another. Utilizing a VAE [17] type architecture, they use Euclidean vector space as the group and define the group action on the data space to correspond to addition in the latent space. This results in group action the data space by enforcing the encoder and decoder to both be diffeomorphic mappings. However, this design choice significantly limits the flexibility as it can only model groups that are isomorphic to a real vector space.
Another line of work that is related to our work is novel view synthesis, where the goal is to take an input image of scene and generate a new image of that scene from novel camera pose. Although our method is not specifically a novel view synthesis model, i.c., it is a method for more general latent group actions, we do demonstrate its effectiveness in novel view synthesis tasks as example of 3D rotation group actions. Therefore, we compare to existing geometry-free free novel view synthesis methods [9, 28]. Unlike geometry-awer methods [18, 27, 35], geometry frree approaches do not necessitate test-time optimization, which is also the case for our method. For example, Sajjadi et al. [25] utilizes transformer architecture which also support multiple view inputs. On the other hand, Dupont et al. [9] introduces equivariance into the latent representations. However, unlike our method, they directly apply group actions the 3D latent volumes as if they are 3D images. This results in higher time complexity despite their relatively small model sizes. Also note that novel view synthesis by definition intended learn the view transformation of the whole scene. While our approach is more flexible and for example can be applied to the scenario where only a foreground object is rotating, while the background stays fixed.
3 Group actions on latent representations
We will consider an autoencoder that takes a data point x E X' and encodes it into latent representa- tion E through the encoder mapping E X + Z. The decoder mapping D Z - X maps a latent representation back into the data space. Typically, the data space, X, and latent space, 2, are real vector spaces, but the mathematical development below does not require this. We will denote the image of D as X" = -m (D) S X, which is the space of all possible reconstructions of the decoder. Furthermore, we consider a group G that acts on the latent space 2 via
a: G x 2 + 2.
When the context is clear, we will denote the group action as g = arg, z). We will also use the notation 0gy 2 + Z to mean (2)2) = agg z). As reminder, group action is required to follow
two rules for all 91 92 E G and all 2 E Z:
2. 2,
92 (g1.2) = (g291) .2.
where G' denotes the identity element.
3.1 Varying and invariant latent representations
In general scenarios, we need to model both factors that are varying with the group action and those that are invariant to the group action. Coming back to the example of taking photos of rotating 3D objects, the background invariant to the rotations. Therefore, we propose to split the latent space Z into a direct product of varying and invariant parts: 2 = Z, x 2, We will write point E 2 as
[zv 25].
where E E, and E Z, represent factors that are varying with and invariant to the group action, respectively. Given a group action g on Z,, we define the group action on Z to be:
9 2 [g. Zoj 24].
3

---

X
K&V
Attention Module
concatenate
X
downsmmpling
module
upsampling
moduul
x M
M
optional module
Input Image
Output Image
FC
Figure 2: Illustration our latent space group action model.
It is clear that the identity element e E G will leave fixed. Next, we check the associativity:
92 (g1 2)
[92- 99-1 Zuj 24]
[(2 91) Zpj 21]
(02 91) 2,
and therefore, this valid group action on 2.
3.2 Latent space group action model
The goal of training is to jointly learn to autoencode a latent representation of our data along with group action those latent factors. For the target group action a, directly compute a(g, 2). The output of this group action is then sent as input to a decoder, D. During training, we look at pair of data, I1, 22 X, at time, whose corresponding latent representations, 21, 22 Z, lie in the same orbit, and we are given E G such that 22 = 21
The loss is simply the reconstruction loss with group actions, where the objective is to minimize:
Lx(x2, D(g 21))
Cx can be any standard reconstruction loss appropriate for data in X, e.g., binary cross entropy (BCE) loss mean square error (MSE), etc. Similarly, we symmetrically add the reconstruction loss of æ1. which gives the final form:
((x1.02) = Lx(12, D(g. 21)) + D(g .2))).
We also find adding LPIPS [36] loss is also helpful for image rendering. It is especially beneficial when the given image only contains very limited information of the latent factor on which the group acts. For example, if rotate 3D object, having single random angle of view only gives us scarce information. In our experiments, we found that the perceptual loss guides the model to render more visually convineing images this circumstance.
3.3 Skip connections and attention
When applying group actions to the latent representation z, we would like to maintain generalizability by acting on itself instead of transforming the latent coordinates done by Dupont et al. [9]. This requires us to sufficiently "mix" spatial dimensions when the given group action affecting the image globally. However, it is well-known fact that this will result in some loss of image details [19]. To address this, we include skip connections our architecture inspired by the U-Net [24].
We skip connect the higher resolution features from the earlier stages of the downsampling path to the corresponding step of the upsampling path. Since the spatial dimensions of the higher resolution features are not mixed adequately, an attention module [29] is applied before concatenating to the upsampling path, where the upsampled feature acting as the query. This gives us the overall framework of our method, as illustrated in Figure 2. The downsampling module can employ either convolutional layers paired with pooling layers or strided convolutions. Conversely, for upsampling, transposed convolutional layers in conjunction with plain convolutional layers can be utilized.

---

Induced group actions on the data space
In this section, we discuss the conditions under which group action on the latent space, Z, will induce group action the output space of the decoder, X' im(()) C X. It is worth mentioning that we don't 't define this as a group action on the full data space, X, because it doesn't 't make sense to consider points in X' that can't 't be reconstructed by the decoder. We start with definition:
Definition 4.1. A decoder D is called consistent with group action of G Z if for any 21, Z such that ( = (222). it is the case that Dig 21) = D(g 22) for any g E G.
Proposition 4.2. Let D be a decoder consistent with a group action a x - Z. Then D induces group action å G x X' X' X' im (D), defined follows. For any I X', there exists E 2 such that (=)) = x. Then a(g, x) = Díg 2). In other words, the following diagram commutes:
2
2
ag
X
Proof. We first note that the induced group action is a well-defined mapping. That is, ā(g.z) does not depend the latent representation of z, precisely because of the consistency condition on D. Next, we show that ā satisfies the properties of a group action. For any 91, 92 E G and x E X", we pick E 2 such that (2),, and have
<(e,2) D(a(e, 2)) D(2)= and
a(g2, ā(g1.)) = ā(g2, D(a(g),))))
D(a(ge, alg2, ((92,) 2)))
D(a(gesh;
a(g2g1,
which are the two properties for be a group action.
The consistency condition of Definition 4.1 is difficult to check directly. The next result shows more intuitive condition that a decoder that can inverted by an encoder satisfies the consistency condition for any group action on its latent space.
Proposition 4.3. Let E: X Z and D Z - X be autoencoder such that E(D(=)) = for all z E Z. Let a : G x Z - Z be group action. Then is consistent with a and thus induces a group action à on X' (as defined in Proposition 4.2).
Proof. The condition B(D(z)) = implies that D is injective. To see this, consider two points 21. 22 E Z such that 21 * 22. Then it must be the case that D(21) * D(22) in order for E(D(21)) 21 * 22 = E (z(2))) Next, from Definition 4.1 it is clear that injectivity of D implies it is consistent
with any group action.
We note that the condition that E(D(=)) for all E Z, i.c., that E be left-inverse of D, does not imply (xE(z)) = æ. In other words, the encoder and decoder are not necessarily (two-sided) inverses of each other, as is the case in [13]. In that work, the two-sided inverse requirement means that and must be vector spaces of the same dimension. For the one-sided inverse condition in Proposition 4.3, Z and X' do not have to be isomorphic sets. For example, if Z and X' vector spaces, Z can have smaller dimension than X.
5 Examples
5.1 2D and 3D rotations
We look at 2D and 3D rotation groups, SO(2) and SO(3). The rotation matrix y E SO(k) acts on Z in the manner described in Section 3.1. As for the group action of SO(k) on Z,, apply given
5

---

rotation matrix to every subset of & dimensions separately. More specifically, we reshape 2. into an k n matrix, with dim(2,)/ Denoting this matrix zny we define the group action on 2 as:
9 9- [vec(g = zs]:
where veo() indicates reshaping the given matrix into vector.
5.2 Image contrast transformations
An image contrast transformation is a diffeomorphic function from [0. 1] to [0. 1] that is applied to each pixel value. We can define two- parameter family of image contrast transformation on the real line, = (ff = 2 x R, as follows. Note an element (a, b) E G represents a scaling by a and shift by b of the real line. The group operation of G is (ai, bi) (a2, b2) (a) a2, bj +a b2). Now, we show that G can also act on the unit interval. Let (a, b) E G and æ E [0, 1] be a pixel intensity, and define
the affine group
(a, = logiit(()) + 6).
The sigmoid function and logit are inverse functions of each other. We prove this is valid group action on [0, 1], where the group being (1) The output of o() is always in [0, 1], and therefore. [0, is closed under the group action. We can model this in the latent space with simpler A group action:
(a, (azz + b; zz]-
5.3 Cyclic group transformations
In additional the continuous groups, our method can also be applied to discrete group actions. equipped with the binary operation
The cyclic group C& is equivalent to the set 2, defined as addition modulo In implementation, this can be represented as the subset of SO consisting of discrete rotations by angles f0, 2F 4 can be incorporated into our model as described in Section 5.1.
k .............
2n(k-1) 1) }. Then this representation of C₁ k. ..... k
Rotated MNIST
Rotated and blocked MNIST
Brain MRI
NMR
Plane in the sky
X1
X2
Figure 3: Data samples used in the experiments. Each column represents pair related by a group action, i.c., I = 22
Experiments
We conduct experiments on five different image datasets. Our models and all the baseline models train on pairs of images. We show some sample pairs from each dataset in Figure 3.
6.1 Datasets
Rotated MNIST dataset is obtained by simply rotating images from MNIST dataset [6] about the center by random angles 0, where 0 ~ (4 2T). Bilinear interpolation is used for sampling. We use the original train-test split and image size of 28 x 28. The group action for this dataset is 2D rotation with the group being SO(2). The ground truth actions between each pair is easily computable from the rotation angles. To ensure that the group action is also the data space, stated in Proposition 4.3, we introduce an additional loss term, || - E()())) [3, where 2 denotes latents encoded from real data. Please see the appendix for more details.
Rotated and blocked MNIST dataset further processed from the rotated MNIST dataset by adding randomly placed 7 7 white square to each image with probability 0.8. The group action is still

---

2D rotation. Moreover, images of a pair either have no squares or have squares at the same locations. In other words, the square is invariant to the group action. This gives us a dataset where the rotation is acted on the latent nector-rrpreeenttatin
of the digits- and not on the whole image.
Brain MRI dataset is derived from the Human Connectome Project [1]]. It consists of 1113 31> brain magnetic resonance images with brain segmentations from FreeSurfer [10]. We took mid- coronal 2D slices for each subject, among which 880 images are used for training and 233 for testing. Then the image contrast transformation defined in Section 5.2 is performed only to the pixels in the brain, leaving the remaining parts (skull, neck, etc.) unchanged. Note that this is not a simple contrast transformation on the whole image. The model needs to identify the brain pixels, and thus, this is again a group action on a latent factor and not the data itself. We randomly sample a = et,t t ~ N (0, 0.25) and b ~ No0 0.25) 100 times for each original image.
Neural 3D mesh renderer (NMR) [14] dataset has been used in multiple previous works in the field of novel view synthesis. This dataset is derived from ShapeNet [3] by rendering each object at 24 fixed views around the object in a cycle. This forms cyclic group, C24. acting on the latent factor- the camera angle. We stick to the original split for training and evaluations.
Plane in the sky dataset is our own rendering of ShapeNet Core [3] airplane objects. Each airplane is put in a real sky background cropped from the SWIMSEG sky image dataset [7]. We uniformly sample 100 random 3D rotation matrices from SO(3) and rotate the plane. Different from the novel view synthesis problem, this results in sets of images with the same sky background and varying plane orientations. We randomly split out 20% the testing set.
6.2 Comparison to baselines
Implementation details and baseline models. All our experiments are implemented with the Py PyTorch [20] package. As illustrated in Figure 2, our encoder consists of several convolutional downsampling modules, while our decoder comprises convolutional upsampling modules. We used skip connections and attention modules for the two rendered 3D objects datasets. For more details of model architectures for each dataset, readers can refer the appendix. The training is performed by randomly sampling pairs of data points from the same orbit. We used MSE for the reconstruction loss, and further added LPIPS loss for rendered 3D objects datasets with a weight of 0.0005.
For the two MNIST derived datasets, we compare to Hwang et al. [13] and Winter et al. [32]. While for the brain MRI set, only compare to Hwang al. [13], since Winter et al. [32] requires tailored equivariant layer for each group and there is no readily available one for the image contrast transformation group. For the two datasets that are rendered images of 3D objects, we compare to two novel view synthesis models - Sajjadi et al. [25] and Dupont et al. [9].
For the encoders of Hwang et al. [13], which encode the group action given each pair, we use almost identical architectures as our encoders. The only difference is that their encoders are variational and require reparameterization. We carefully follow their work to implement the decoders, which are built upon Glow [16] type of normalizing flow architectures. All the other baselines have official or officially recognized implementations.
To evaluate the model's 's ability in learning group actions accurately, we compare the predicted D(g 2) with the ground truth image. Since Hwang et al. [13] encodes group actions based pairs only, we present another pair of samples, 23 and I4, s.t. Wi = g g.22g, 2g, for their models to encode g. In our experiments, we find their model performs best when given 2y and v44 having the same absolute angles, or the same (a, b) w.r.t. the original images, as 21 and 22 respectively.
Qualitative results. The sample reconstructions for the first three datasets are shown in Figure 4. As we can see, Hwang et al. [13] demonstrates problems learning correct rotation angles for both MNIST derived datasets.. Visually. Winter et al. [32] and our models exhibits comparable performance on the rotated MNIST, indicating the capability of our models to model group actions on X' as well. As for the rotated and blocked MNIST, their model attempts to learn rotation invariant representations, leading to challenges correctly modeling the blocks. At times, the model tends to align blocks with the digit structures, as the rotation is acted on the digit, as illustrated in the first column of the sample outputs. While for the brain MRI dataset, both models show impressive performance, and is difficult to determine a noticeable advantage visually.
7

---

2
5
Inputs
Inputs
Winter
2
Q
8 5
Hwang ct al.
Hwang
0
Ours
Ours
Q
3
5
4
Ground truth
3
5
4
Ground truth
(a) Rotated MNIST
(b) Rot. & bl. MNIST
(C) Brain MRI
Figure Sample reconstructions on MNIST derived datasets and brain MRI dataset
Inputs
Dupont
X
X
Sajjadi al.
Ours
X
y
Ground
truth
(a) NMR
(b) Plane the sky
Figure Sample reconstructions on NMR dataset and plane in the sky dataset
The sample reconstructions for the two rendered 3D objects datasets are shown in Figure 5. As NMR novel view synthesis and baseline models are designed for this task, we can see they both render reasonable reconstructions with Sajjadi et al. [25] giving sharper images. Our model also correctly learns view angle changes and even captures image details better in some cases. While for the plane in the sky dataset, which is not novel view synthesis dataset, we can see both baselines struggle give convineing results. Dupont al. [9] has relatively better reconstructions while Sajjadi et al. [25] sometimes miss the plane orientation significantly. Our model renders much higher quality images with correct orientations and better details.
Table 1: Quantitative results on MNIST derived datasets and brain MRI dataset
Rotated MNIST
Rot. & bl. MNIST
Brain MRI
1PSNR
1SSIM
1PSNR
1SSIM
1PSNR
1SSIM
Winter al. [32]
21.97
0.874
14.05
0.586
NA
NA
Hwang et al. [13]
15.29
0.992
10.19
0.990
27.43
1.000
Ours
26.07
1.000
23.55
1.000
35.99
1.000

---

Table Quantitative results on 3D objects rendered datasets
NMR
Plane in the sky
1PSNR
1SSIM
LLPIPS
PPSNR
1SSIM
JLPIPS
Dupont et al. [9]
26.91
0.899
0.091
24.25
0.773
0.239
Sajjadi et al. [25]
27.87
0.912
0.066
23.53
0.489
0.280
Ours
28.91
0.947
0.050
25.24
0.821
0.112
Quantitative results. We chose peak signal-to-noiss ratio (PSNR) and structural similarity index measure (SSIM) as quantitative metrics for the first three datasets and present resulting values in Table 1. Our models consistently achieved the best results across all three datasets and both metrics. For the NMR dataset and plane in the sky dataset, we further compute LPIPS (VGG) in addition PSNR and SSIM. The results reported in Table 2. Our models also achieves best results consistently, which aligns with the qualitative results. We can also observe that between the two baselines, Sajjadi et al. [25] is performing better on NMR dataset, while Dupont al. [9] performing better on the plane in the sky dataset. Finally, it's also worth noting that we checked how well Proposition 4.3 satisfied with the additional training loss encourage it. The average reconstruction L2 distance is 0.304. Comparing to the standard deviation of & over the dataset being 4.043, we conclude that this property is approximately met.
Discussion. It is worth noting that the model by Winter et al. [32] is specifically tailored for group actions the data space X, which accounts for the performance discrepancy on the rotated and blocked MNIST, where the block does not rotate with the rest of the image. Additionally, even the rotated MNIST is not precisely group action on X' given the interpolations occurring during rotation. Their original experiment utilized blurred rotated MNIST, different from our dataset, as measure to mitigate this problem. As for Hwang et al. [13], it models the group Euclidean space R4 under addition in the latent space. We can see that their model especially struggles on the rotation groups, (2),), which are not Euclidean spaces. Furthermore, the normalizing flow architecture used for encoder/decoder enforces these mappings to be diffcomorphisms, resulting in group action on X. which accounts for its poor performance on the blocked MNIST.
Dupont et al. [9] incorporates group equivariance in the latent space by transforming the latent coordinates. Although this exactly models novel view synthesis task, it is too rigid for more general scenarios, as the plane in the sky dataset. Furthermore, we can see that is not very good at capturing details from the results on NMR dataset. However, the introduction of equivariance might have helped modeling more difficult 3D rotations of the planes, giving it better performance compared to Sajjadi et al. [25]. On the other hand, Sajjadi al. [25] transformer type of model that does not incorporate group equivariance. It is better at capturing details and rendering sharp images as seen in results on NMR dataset. However, fails on the plane dataset, which more challenging task than most novel view synthesis datasets, given full 3D random rotations and single input view.
Table 3: Quantitative results of ablation study on NMR dataset
PPSNR
1SSIM
[LPIPS
Ablation 1
27.31
0.933
0.080
Ablation 2
26.96
0.930
0.063
Ablation 3
28.09
0.941
0.072
Ours (full)
28.91
0.947
0.050
6.3
Ablation study
We also performed an ablation study on the NMR dataset to explore how skip connections paired with attention modules and the addition of LPIPS loss impact our model performance. We define the following three ablation models: Ablation
our model without the skip connection and trained without the LPIPS loss; Ablation 2 - our model without the skip connection; Ablation 3 our model trained without the LPIPS loss. The results are listed in Table and some samples are shown in Figure 6. Combining skip connections with LPIPS loss yields the best performance across all metrics,

---

Ablation
Ablation 2
Ablation
Ground
Inpuls
Full
truth
}
Figure 6: Samples from ablation models and our full model.
significantly enhancing image details. Skip connections alone provide a performance boost, but adding LPIPS loss without skip connections offers minimal improvement. Notably, even without skip connections or LPIPS, our model accurately captures object orientation and performs comparably to baseline models, highlighting the robustness of our latent group action modeling strategy.
S
3
Inputl
Inpuil
2
2
2
Input2
Inpui2
S
3
Swapped1
Swapped1
2
2
Swapped2
Swapped2
(a) Rotated and blocked MNIST
(b) Plane in the sky
Figure 7: Samples generated by swapping invariant and varying parts of latent representations.
We further investigate how varying and invariant information is captured by swapping the s, and zi components of the latent representations between inputs and decoding the results. Some examples are shown in Figure 7. Interestingly, the model generalizes to new combinations of varying factors (e.g., digit and plane shapes) and invariant factors (e.g., the white block and the sky), despite not being explicitly trained in such fashion.
7 Conclusion
In this paper, we propose novel approach of learning group actions on latent representations. Our experimental results demonstrate that our method can effectively model broader range scenarios than existing models of group actions on the data space. In addition to being able to model group actions in the latent space, we show both theoretically and empirically that our strategy is also capable of modeling group actions on the data space. Furthermore, we achieve state of the art performance on the geometry-free novel view synthesis task, and we outperform previous approaches to learning group actions in more general cases. We note that our model requires ground truth group actions during training, which might not available some cases. We leave it as future work to apply our method in semi-supervised unsupervised manners.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by NSF Smart and Connected Health grant 2205417.
References
[1] Ahmed, Y. Bengio, H. van Seijen, and A. C. Courville. Systematic generalisation with group invariant predictions. In 9th International Conference on Learning Representations, ICLR, 2021.
[2] D. Bortoli, E. Mathicu, M. Hutchinson, J. Thornton, Y. W. Teh, and A. Doucet. Riemannian score-based generative modelling. In Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS, 2022. 2
[3] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang. Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report 12Xiv:1512.0341
[cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015. 7
[4] T. Cohen and M. Welling, Group equivariant convolutional networks. In Proceedings of the 33nd International Conference on Machine Learning, ICML, volume 48, pages 2990-2999, 2016. 1,2
[5] S. Cohen and M. Welling. Steerable CNNs. In 5th International Conference on Learning Representations, ICLR, 2017. 2,
[6] L. Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2966)1141112,, 2012. 6
[7] S. Dev, Y. H. Lee, and S. Winkler. Color-based segmentation of sky/cloud images from ground- based cameras. IEEE Journal of Selected Topies in Applied Earth Observations and Remote Sensing, 10(1):231-26)
2016." 7
[8] N. Dey, Chen, and S. Ghafurian. Group equivariant generative adversarial networks. In 9th International Conference Learning Representations, ICLR, 2021. 1,2
[9] E. Dupont, M. B. Martin, A. Colburn, A. Sankar, J. Susskind, and Q. Shan. Equivariant neural rendering, In International Conference on Machine Learning, pages 2761-2770. PMLR, 2020.
3, 4, 7, 8, 17
[10] B. Fischl, D. H. Salat, F. Busa, M. Albert, M. Dieterich, C. Haselgrove, A. Van Der Kouwe, R. Killiany, D. Kennedy, S. Klaveness, et al. Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Newron, 33(3):341 2002. 7
[11] M. F. Glasser, S. N. Sotiropoulos, J. A. Wilson, T. S. Coalson, B. Fischl, J. L. Andersson, J. Xu, Jbabdi, M. Webster, R. Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 10:105-124 2013. 7
[12] E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling. Equivariant diffusion for molecule generation in 3D. In International Conference on Machine Learning, ICMI., volume 162, pages 8867-8887, 2022. 1,2
[13] G. Hwang, J. Choi, H. Cho, and M. Kang. Maganet: Achieving combinatorial generalization by modeling group action. In International Conference on Machine Learning. ICML, volume 202, pages 14237-14248, 2023. 3, 5, 7, 8, 9, 17
[14] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907-3916. 2018. 7
[15] H. Keurti, H.-R. Pan, M. Besserve, B. F. Grewe, and B. Schölkopf. Homomorphism AutoEn- coder learning group structured representations from observed transitions. In International Conference Machine Learning, ICML, volume 202, pages 16190-16215, 2023, 1
11

---

[16] D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems, NeurlPS, pages 10236-10245, 2018. 7
[17] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In 2nd International Confer- ence Learning Representations, ICLR, 2014.
[18] K.-E. Lin, Y.-C. Lin, W.-S. Lai, T.-Y. Lin, Y.-C. Shih, and R. Ramamoorthi. Vision transformer for nerf-based view synthesis from single input image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 806-815, 2023. 3
[19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3431-3440, 2015.
[20] A. Oring, Autoencoder image interpolation by shaping the latent space, PhD thesis, Reichman University (Israel), 2021.7 7
[21] Y. Park, O. Biza, L. Zhao, J.-W. Van De Meent, and Walters. Learning symmetric embeddings for equivariant world models. In International Conference on Machine Learning, pages 17372-17389. PMLR, 2022.
[22] R. Quessard, T. D. Barrett, and W. R. Clements. Learning disentangled representations and group structure of dynamical environments. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS, 2020. 1,2
[23] D. Romero, E. Bekkers, Tomozak, and M. Hoogendoom. Attentive group equivariant convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, ICML, pages 8188 (199,,89992001 2020.
[24] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisee InterventionMMICC. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015. 4
[25] M. Sajjadi, H. Meyer, Pot, U. Bergmaan, Greff, N. Radwan, S. Vora, M. Lučić, D. Duckworth, A. Dosovitskiy, et al. Scene representation transformer: Geometry- freeett---reeeeee novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6229-6238, 2022. 7, 17
[26] V. G. Satorras, E. Hoogeboom, and M. Welling. E(n) equivariant graph neural networks. In Proceedings of the 38th International Conference Machine Learning, ICML, volume 139, pages 9332-9332, 2021. 2
[27] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-awart neural scene representations. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, NeurIPS, pages 1119-1130, 2019. 3
[28] V. Sitzmann, S. Rezchikov, B. Freeman, J. Tenenbaum, and Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Advances in Neural Informa- tion Processing Systems, volume 34, pages 19313-19325, 2021. 3
[29] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 4, 16
[30] D. Wang. J. Y. Park, Sortur, L. Wong, R. Walters, and R. Platt. The surprising effectiveness of equivariant models domains with latent symmetry. NeurIPS 2023 Workshop on Symmetry and Geometry in Neural Representations, 2023. 2
12

---

[31] M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen. 3D steerable CNNs: Learning rotationally equivariant features volumetric data. In Advances in Neural Information Process- ing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS, pages 10402-10413, 2018. 2
[32] R. Winter, M. Bertolini, T. Le, Noé, and D. Clevert. Unsupervised learning of group invariant and equivariant representations. In Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS, 2022. 3, 7, 8, 9, 17
[33] T. Yang, X. Ren, Y. Wang, W. Zeng, and N. Zheng. Towards building A group-based unsuper- vised representation disentanglement framework. In The Tenth International Conference on Learning Representations, ICLR, 2022. 1
[34] Yim, L. Trippe, D. Bortoli, E. Mathieu, Doucet, Barzilay, and T. Jaakkola. SE(3) diffusion model with application to protein backbone generation. In International Conference on Machine Learning, ICML, volume 202, pages 40001-40033 2023. 1,2
[35] A. Yu, Ye, M. Tancik, and A. Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1578-4587 2021. 3
[36] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features a perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
13

---

A Model Architectures
A.1 MNIST derived datasets
Table 4: The architecture of the encoder two MNIST derived datasets
Downsampling module (32 channels)
Downsampling module (64 channels) Downsampling module (64 channels)
Downsampling module (64 channels)
Flattening
Fully connected (256 neurons)
ELU activation
Fully connected (24 neurons)
Table 5: The architecture of the downsampling module two MNIST derived datasets
3 x 3 convolution with stride 2
ELU activation
Table 6: The architecture of the decoder on two MNIST derived datasets
Fully connected (256 neurons)
ELU activation
Fully connected (256 neurons)
ELU activation
Unflattening
Upsampling module (64 channels)
Upsampling module (64 channels)
Upsampling module (32 channels)
Upsampling module (1 channel)
Table 7: The architecture of the upsampling module on two MNIST derived datasets
3 X 3 transposed convolution with stride 2
ELU activation (Sigmoid activation for the last module)
14

---

A.2
Brain MRI dataset
Table 8: The architecture of the encoder on the brain MRI dataset
Downsampling module (32 channels)
Downsampling module (64 channels)
Downsampling module (64 channels) 2D batch normalization
Downsampling module (64 channels)
Downsampling module (64 channels)
3 x convolution (64 channels)
Leaky ReLU activation
3 x convolution (64 channels)
Table 9: The architecture of the downsampling module the brain MRI dataset
3 x 3 convolution with stride 2 Leaky ReLU activation
Table 10: The architecture of the decoder on the brain MRI dataset
3 x convolution (64 channels)
Leaky ReLU activation
3 x convolution (64 channels)
Leaky ReLU activation
Upsampling module (64 channels)
Upsampling module (64 channels)
2D batch normalization
Upsampling module (64 channels)
Upsampling module (32 channels)
Upsampling module channel)
Table 11: The architecture of the upsampling module the brain MRI dataset
x 3 transposed convolution with stride 2
ELU activation (Sigmoid activation for the last module)
15

---

A.3 3D objects rendered datasets
For these datasets, we employ skip connections and attention modules. For the attention model, we use the standard architecture from Vaswani et al. [29], which is not listed here. The only difference is that instead of self-attention, we have the input from the encoder acting keys and values while the input from the decoder acting as queries. For more intuitive illustration, see Figure 2.
Table 12: The architecture of the encoder two 3D objects rendered datasets
Convolutional block (64 channels)
Downsampling module (128 channels)
Downsampling module (256 channels) Skip connection, imput to attention module Downsampling module (512 channels) Downsampling module (1020 channels)
Table 13: The architecture of the downsampling module two 3D objects rendered datasets
2 x 2 max pooling
Convolutional block
Table 14: The architecture of the convolutional block on two 3D objects rendered datasets
3 x convolution
2D batch normalization
ReLU activation
3 x convolution
2D batch normalization
ReLU activation
Table 15: The urchitecture of the decoder on two 3D objects rendered datasets
Upsampling module (512 channels)
Upsampling module (256 channels) with skip connection Upsampling module (128 channels)
Upsampling module (64 channels)
Channel-wise fully connected
Sigmoid activation
Table 16: The architecture of the upsampling module (without skip connection) on two 3D objects rendered datasets
2 2 nearest upsampling
convolution
2D batch normalization
ReLU activation
Convolutional block
16

---

Table 17:
The architecture of the upsampling module (with skip connection) on 3D objects rendered datasets
2 x nearest upsampling
3 X 3 convolution
2D batch normalization
ReL.U activation
skip connection, input to attention module attention module output, concatenate
Convolutional block
B Number of Learnable Parameters
We also compare the numbers of learnable parameters across our models and baselines in Table 18. As can see, our model is relatively small which indicates did not achieve high performance by simply using complex architectures. The only smaller baseline model is Dupont et al. [9] on the 3D rendered datasets. However, is not any more efficient to train because of the rotation transformations on the latent coordinates requires resampling.
Table 18: Comparison of learnable parameter numbers
MNIST derived datasets
Brain MRI dataset
3D rendered datasets
Winter et al. [32]
1 1.22M
Hwang et al. [13]
3.06M
9.33M
Dupont et al. [9]
11.2M
Sajjadi et al. [25]
73.9M
Ours
0.33M
0.41M
32.3M
C Training details
Our architecture trained A100 GPU with batch-size of 256 using the Adam optimizer. The learning rate is 0.0001. We randomly split 1/8 of the training set as the validation set. All models selected based on the best validation loss. The architecture and the training was implemented in PyTorch and the code will made available upon publication.
For the training the rotated MNIST dataset, added the additional loss of || E(D(2)) with a weight of 0.0001 to enforce the assumption of Proposition 4.3 that the decoder is consistent with the group action. While this loss term does not improve or worsen the reconstruction performance on the test set, it induces a group action on the data space (theoretically, if it were to be exactly zero). Exploring the implications of ensuring group action in the data space, rather than only in the latent space, is left for future work.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: We included some theoretical analysis on our methods and its relation to existing methods. And conducted extensive experiments show how our methods apply to various scenarios and drew comparisons to existing methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: our conclusion, we talked about our limitations, which is reliance on ground truth group annotations. We leave this part as our future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[Yes]
Justification:
Section 4 is dedicated for this.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In addition to the description in the paper, we included more detailed model architectures the appendix. All our datasets can be derived from public datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The data all derived from publicly available datasets. We believe we provided enough details. We will make the code available in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpiccccceecce c gugdediessssse CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All our splits are random except for the NMR dataset which we followed original split. We believe the split won't It substantially affect results.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We did not perform this.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
We believe it is irrelevant to our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : //nuurrpss... cc httpppplllccchciiiuuiit
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "uqKK9QrUoS", "reviewer": "Reviewer_yBaK", "summary": "The paper introduces a new method to learn group structure such as SO(3) rotations on data by representing the group actions in a latent space. The authors propose an autoencoder-based method to represent this latent space, and demonstrate how it can represent equivariance to latent factors.", "questions": "Most of the my questions are in the section above. I am curious to see evidence demonstrating how successful the latent space is at disentangling invariant and varying parts of the input data. \n\nAdditionally, do the residual connections interfere with the group structure of the latent space at all?", "limitations": "The paper did not discuss limitations besides the fact that the method requires ground truth labels for group actions.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well motivated. In many real world applications, it is important model equivariant structure of data which is not perfectly equivariant to group actions. It makes sense to decompose data into varying and invariant parts. \n\nFor 2D image data, the method shows a significant improvement over prior work on modeling group actions. The rotated and blocked MNIST dataset demonstrates an example of how the method is still able to model rotation equivariance even if the training data is not perfectly equivariant. \n\nOn the 3D view synthesis datasets, the authors also achieve better image quality on the NMR dataset (which is fully equivariant), and the new Plane in the Sky dataset (which has both rotation varying and invariant parts).", "weaknesses": "I was surprised to find no experiments or analysis on the latent space of the autoeconder. From the experiments, it seems that the paper only studies the outputs of to the auto encoder without any experiments on the latent space itself. It is hard to tell to what extent the improved metrics are due to a better neural network architecture versus the modeled group action. \n\nTo support the paper's central motivation, there should be some experiments that demonstrate that the model learns a meaningful latent space. There should be evidence to demonstrate that the invariant and varying parts of the input images are disentangled properly. For example, some experiments can be\n- Visualizing the output of D(z) from randomly sampled z vectors. Does it output reasonable images?\n- Demonstrating to what extent the decoder D is consistent, as defined by Def 4.1. \n- Demonstrating if or if not E(D(z)) for all z, one of the assumptions of Prop 4.3. \n- Visualizing the invariant and varying parts of the latent space. Can you swap the varying and invariant parts of two latent vectors? For example, if you take two Plane in Sky images and swap the varying and invariant parts of their latents, then the decoded images should show the plane of one image with the background of the other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate the time and effort you have dedicated to reviewing our paper. We will address the weaknesses and questions you raised below.\n\n**Weakness**\n- We note that we actually used multiple architectures in the experiments, sometimes with \"plain\" CNNs (MNIST and MRI experiments), and sometimes with the skip connections and attention mechanism (3D models). While skip connections with attention is proposed to improve the image rendering quality, it is not the decisive factor of our method. As listed in Appedix A, for our two MNIST rendered datasets and the brain MRI dataset, we used plain deep CNNs without skip connections or attetnion mechanism. While we did use it for the two 3D object rendered datasets, we conducted ablation studies in Section 6.3 that shows our model results in state-of-the-art performance even without this additional technique. We also included some samples as qualitative results in the PDF attached to our global rebuttal response. In addtion, we compare the number of learnable parameters in Appendix B, which shows that our method did not win by simply boosting up the architecture complexity.\n- Our model is not a generative model. None of the competing models we compared to is generative either. They are image-to-image models. Therefore, there is no straightforward way to randomly sample latent codes. A generative model, e.g., diffusion model, could be trained in the latent space, but this is outside the scope of the paper. However, the other proposed experiments are conductable.\n- We checked Prop. 4.3 on our rotating MNIST dataset, since it is an example of group action on the data space (other experiments are group actions on the latent space, but not on the data space, so Prop. 4.3 will not hold). The average $z$ reconstruction L2 distance is 0.304. Comparing to the standard deviation (using L2 distance) of $z$ over the dataset being 4.043, we can conclude that this property is approximately met. As we proposed in Prop. 4.3, this indicates the autoencoder is consistent (Def. 4.1).\n- We tried rendering images on blocked MNIST and planes with swapped invariant representations. For the plane dataset, this includes the skip connection with attention module. Some samples are shown in the PDF we attached to the global rebuttal response. The results are what one would expect: The number and the orientation are unchanged, while the block is swapped; the plane orientation and overall shape are unchanged, but the sky background and the details such as colors are swapped. The model is able to combine group invariant and varying information to render new image, even though not sepcifically trained in such fashion.\n\n**Questions**\n- As shown in samples of ablation models from attached PDF, the skip connections only help with image details and sharpness. Also, we reiterate that models on MNIST datasets and brain MRI dataset do not have skip connections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PSQHVUSmU3", "reviewer": "Reviewer_tMkz", "summary": "This paper focus on learning group actions in latent space instead of data space. The group action takes effects between the encoder and decoder in an auto-encoder structure. Several tasks with different group actions are evaluated in the experiments.", "questions": "Please see the weaknesses.\n\nBesides, I would like to know the reason why the proposed method is better at PSNR on Brain MRI dataset compared to [13]. Firstly, this dataset does not contain any 'occlusions', which means it may be difficult to demonstrate the advantages of the proposed method on this dataset. And the authors also find 'it is difficult to determine a noticeable advantage visually'. Then why the proposed method shows much better performance on PSNR?", "limitations": "Limitations are discussed in conclusion section.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized and the core ideas are explained clearly.\n\n2. The results on rotated and blocked MNIST are impressive, which demonstrated that in the latent space, the variant and invariant parts are easier to be disentangled.", "weaknesses": "1. The novelty of learning group action in latent space is not so strong, as the operation on latent space is not rare in lots of other tasks, such as disentangle different factors in generation model.\n\n2. The reason of framework building is not well explained in section 3.3. It is easy to understand directly sending high resolution to the decoder help to keep image details. However, the reason of applying attention operation to the skip connection is not explained, which is also not included in the ablation study.\n\n3. The ablation study results will be more convincible if the qualitative results with/without different parts are compared. For example, LPIPS is helpful for image rendering. Without LPIPS, are the numbers rotated correctly (even the image may not be as clear as the full model)? Or is the model able to separate the numbers and the blocks without LIPIS? These are not revealed from the quantitative results but can be easily demonstrated by the visualizations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate the time and effort you have dedicated to reviewing our paper. We will address the weaknesses and questions you raised below.\n\n**Weakness**\n1. While there are some works that explore models with group actions in latent representations, the novelty of our work is that it is the first to model latent group actions that manifest as acting on some features in the data, while leaving other features invariant. Furthermore, we are the first to generalize to handle multiple types of groups without the need for group-specific neural layers. We have referenced all of the work we are aware of regarding group actions in latent representations. If there are more related works in this specific field, we would appreciate it if you could kindly point us to them. We would point out that most work in disentanglement of latent representation factors does not involve group actions at all, and thus does not diminish the novelty of our work.\n2. We tried to explain the reason to include attention module in line 132-134. However, it might be not clear enough. For more intuitive explanation, we can imagine rotating a plane so that its head goes from the left side to the right side of the image. Now, if we directly perform skip connection, the plane head will be concatenated to the left side, and cannot affect the right side of the image during the remaining upsampling path. Therefore, ideally we would like to concatenate the patch that contains plane head to the correct target position. We rely on the attention mechanism to perform this matching. We did not perform ablation study on attention module alone, because skip connection by itself does not benefit performance theoretically.\n3. We agree that qualitative results will be nice to have. We only excluded them because of the page limit. We reported some samples in the PDF attached to our global rebuttal response. It shows that both LPIPS and skip connection contribute to details and sharpness of images, within which skip connection improves the performance more prominently. However, even without either of them, the model can still learn correct rotations. We would also like to clarify that only our two 3D object rotated dataset is trained with LPIPS and skip connections paired with attention. Therefore, two MNIST rendered datasets are trained merely with reconstruction loss defined in line 119.\n\n**Questions**\n- While occlusions do cause the group action to no longer be on the data space, it is not the only case where this happens. For the brain MRI dataset, the image contrast transformation is only happening to pixels inside the brain, and not the entire image; and which pixels are in the brain is specific to each image. Therefore, it is no longer a group action on the data space, only on the latent space (where the model must learn which pixels are inside the brain). This would explain why we have better PSNR results. As for why this is the case given the similar qualitative appearance of the results, this is most likely due to the difficulty for human's to perceive possibly subtle contrast differences in the images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkpbFreFHE", "reviewer": "Reviewer_GMqP", "summary": "This paper introduces an approach to modelling group actions using autoencoders, particualrly by learning these actions in the latent rather than observed data space. The authors argue that this allows modelling a broader range of real-world scenarios, and does not require particular layers - however, it does require specific architectural changes (e.g., specific skip connections, and  attention modules before concatenating upsampling path).  In practice, it seems that the main idea of the paper is to enforce a sort of cycle consistency on the decoder output (e.g., bidirectional consistency), with inputs the group action applied to the latent representation. The authors demonstrate the effectiveness of their approach through  experiments on  image datasets, covering a range of group actions including. Results appear promising, and good with respect to compared work.", "questions": "How sensitive is the method to hyperparameter choices? e.g. latent dimensionality - split between invariant and variable part\n\nHow does the model scale/what is the computational overhead?\n\nan example that measures, besides similarity metrics, if the resulting representations/data have truly captured the group symmetry (e.g., measuring variability of output/representations given different inputs)", "limitations": "they have", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The ablation study on the NMR dataset (Table 3) provides valuable insights into the contribution of different components (skip connections, LPIPS loss) to the model's performance.\n\nThe approach is simple, and shows promise in transfer learning, few-shot learning, and domain generalization", "weaknesses": "As the authors identify, the model requires ground truth group actions during training\n\nThe authors claim that this method does not require group-specific layers, which it indeed doesn't - however the authors propose a specific architecture to show best results\n\nThe main idea behind this method is basically in the proposed loss function, which is similar to reconstruction-based cycle consistency losses\n\nThe method relies on the loss function to learn these symmetries from data, requiring also ground truth. There are many methods that can attempt to do so if data exists", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate the time and effort you have dedicated to reviewing our paper. We will address the weaknesses and questions you raised below.\n\n**Weakness**\n- We agree that requiring ground truth is a limitation, as we discuss in the paper. However, we would also like to point out that most of the existing works that is applied to model latent group actions, regardless of addressing the problem in terms of group actions or not, are also supervised. This includes both of our comparison models for the two 3D rendered datasets.\n- We note that we actually used multiple architectures in the experiments, sometimes with \"plain\" CNNs (MNIST and MRI experiments), and sometimes with the skip connections and attention mechanism (3D models). While skip connections with attention is proposed to improve the image rendering quality, it is not the decisive factor of our method. As listed in Appedix A, for our two MNIST rendered datasets and the brain MRI dataset, we used plain deep CNNs without skip connections or attetnion mechanism. While we did use it for the two 3D object rendered datasets, we conducted ablation studies in Section 6.3 that shows our model results in state-of-the-art performance even without this additional technique. We also included some samples as qualitative results in the PDF attached to our global rebuttal response.\n- While there is some similarity between our proposed loss term and cycle consistency losses, we would repeat that the novelty of our work is that it is the first to model latent group actions that manifest as acting on some features in the data, while leaving other features invariant. As stated earlier, there are many methods trained with such data, including two novel view synthesis models we compare to in the paper and outperform. As the experimental results show, we believe our model is capable to handle more general types of group actions.\n\n**Questions**\n- Our hyperparemeters are manually tuned on individual valid sets. The numbers of learnable parameters are compared in Appendix B, which indicates our model did not win by overly complicated architecture. We also reiterate that our base architectures are very standard deep CNNs. The only optional new technique is the skip connection with attentions.\n- We have some training details in Appendix C. With this setup, our model was trained approximately 20-40 hours depending on the datasets.\n- We agree that more innovative measurements of how well captured the group symmetry is would be a useful contribution. However, we think this is a quite difficult and subtle property to measure in the output data itself (e.g., how do you know that the variation in an image set is due to a 3D object rotating correctly?). This is a topic for future work. We would note that all of the competing previous works settle for qualitative results and similarity metrics just as we do.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lWaLwb8sCw", "reviewer": "Reviewer_vFp4", "summary": "Learning group actions on latent representations\n\nAbstract\nThe work’s contributions are clear from the abstract.\n\nIntroduction\n\nLines 13-18: Group actions are explained simply and intuitively but the less familiar reader may benefit from a basic example beyond reference to geometric transformations, such as something as simple as an image inversion.\n\nLines 19-30: A simple, elegant, clear example of the problem that the authors are trying to address is covered with a clear distinction between the digit (latent factor) and the image (input data). This really helps the audience to understand the problem and why it is important.\n\nLine 33: Is this something that has been studied in the past by the authors or another group? Is there anything in learning the group actions on the latent space that would or should necessitate learning the same group actions on the data space other than explainability in some cases? In the clear example of the rotating 7 they are both clearly “rotation”. I would need to be more creative to think about a distinct group action on the data that is not present in the latent representations or vice versa. I think the authors probably have some in mind. However, I also understand that the authors are not saying the data and latent group actions need to be exclusive, just that they do no need to be the same either.\n\nLine 36: I understand that having group-specific layers for encoding could limit the architecture of the AE and could limit expressivity. However, is there not an explainability benefit as well to having group-specific layers in the case where several complicated group actions are acting on the latent factors? I am thinking about something like NIT (https://proceedings.neurips.cc/paper/2018/hash/74378afe5e8b20910cf1f939e57f0480-Abstract.html) but for learning group actions instead of learning separable functions on input data akin to a generalized additive model.\n\nRelated works\n\nLine 62: I think it helps any reader that does not do research in group actions to introduce what some of the group actions (here E(3)) mean/do without having to consult references or web search.\n\nOverall the Related works does a great job of summarizing past work on group actions, impacts, and limitations. The differences between the authors’ work and [9] are appreciated and described well.\n\nGroup actions on latent representations\n\nLines 103-107: The notation is made very clear. Group action requirement is defined for the unfamiliar reader.\n\nLines 108-110: The math holds. I would like to hear more about the intuition on representation of the latent space as a product of the varying and invariant parts. I imagine that the latent space can have components that are both varying and invariant to a group action, such as the example given with 3D images. However, it seems that there would be other groups actions present that are fully varying with respect to a group action. In this case we could have z_j varying to group action A and invariant to group action B. Please note whether this is simply a misunderstanding in the notation relative to these representations.\n\nLatent space group action model\n\nLines 111-124: The model formulation is clear. I might go one step further for expanded reach to a broad audience to say something very simple to summarize Lines 115-116 like “z2 is the result of taking group action G on z1” as “lie in the same orbit” might not be explicit enough for all readers. Nothing is lacking in the math for the average reader to follow along. \n\nSkip Connections and attention\n\nLines 126-137: The authors highlighting attention and skip connection as optional shine a light on how their overall approach is relatively architecture agnostic, which is great.\n\nInduced group actions on the data space\n\nLines 139-169: Propositions check out. The diagram helps the reader. Lines 164-169 are helpful as well. \n\nExamples\n\n2D and 3D Rotations\nLines 171-172: 2D/3D rotation as matrix operations (transformations) are described clearly.\n\nImage contrast transformations\nLines 173-174: It has been some time since I thought about image contrast transformation but the authors make it very clear. \n\nCyclic group transformations\nLines 174-179: Again, clearly described. \n\nExperiments\n\nLines 189-193: Highlighting to the reader how this is a latent space group action not a data group action by keeping the occlusion square fixed and not rotating with the image is key. Same for MRI coronal 2D image contrast group action only on the brain and not the extra-dural tissue.\n\nLine 224: I think authors meant to say “they use almost” when talking about Hwang’s encoder.\n\nFigure 4: I assume these are test set images but I would label as such in the Figure 4 caption\n\nQuantitative results\n\nLine 251: Choice of PSNR and SSIM are justified\n\nTable 3: I like the addition of the ablation study. It shows that skip connection and LPIPS helps but that the major gains are agnostic to fine tweaks to the modeling approach to learn latent group actions. This is a good thing.", "questions": "Please see weaknesses above. I would like to know more about how g is initialized or further clarification if I misunderstood the operation in the text vs what is shown in Figure 2.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The problem is clear. Much work has been done to study group action on data but limited work has been done looking at group actions on latent representations. The authors describe why this is important, especially in the era of generative AI for everything. \n\nResults are excellent. A nice surprise is that the author's work, despite focusing on group actions on latent representations, actually does just as well or better on group actions on data as compared to prior work in this area, showing it is invariant to the level of the group action (data vs latent).", "weaknesses": "One thing missing or that I could not find in the main manuscript or paper is an explicit description of how g is initialized. For example, in the 2D rotation case, we need to learn g (reshaped in matrix form as a rotation matrix). The authors note that \"For the target group action α, we directly compute α(g, z)\". I have seen similar (unpublished) work looking at learning the transformation matrix for a group action in the data space with a similar approach but that learning g was initialization sensitive. Clearing this up and making it more clear in Figure 2 that this is multiplication would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very detailed and insightful review. We appreciate the thorough understanding you demonstrated and the thoughtful feedback you provided. We are grateful for the time and effort you invested. We will revise the unclear sentences and captions the reviewer kindly pointed out.\n\nAs for the question you raised, we actually did not learn high dimensional $g$. We discuss about this in the paper, such as in Section 5.2, but it might be not clear enough. Taking the 2D rotation as an example, we reshape the $z_v$, the varying part of the latent representition, into $k\\times 2$ matrix, then do matrix multiplication with the actual 2D rotation matrix $g$. In our preliminary experiments, we experimented with embedding it into high dimensional matrix, but found that none of the embedding techniques work better than simply using the original rotation matrix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the valuable feedback provided by all the reviewers. We are grateful for the time and effort you dedicated to our paper. To provide further clarity, we have included a PDF with additional samples. Please refer to our individual responses for detailed information on each review. We look forward to further comments from reviewers.

Meta-review: This paper introduces a new approach to model group actions.
After the author-reviewer discussion period, this paper received an average score of 5.5, with one negative score of 4.
Since the reviewer with such a negative score did not engage in the discussion, I have checked the author's rebuttal carefully.
I agree that the author has addressed the concerns.

In conclusion, the AC recommends accepting this paper in NeurIPS'24.
Decision: Accept (poster)