Unsupervised Object Representation Learning using Translation and Rotation Group Equivariant VAE
Authors: Alireza Nasiri
Tristan Bepler
Venue: NeurIPS 2022 Accept (2022)
Keywords: deep learning
variational inference
representation learning
group equivariance

Abstract: In many imaging modalities, objects of interest can occur in a variety of locations and poses (i.e. are subject to translations and rotations in 2d or 3d), but the location and pose of an object does not change its semantics (i.e. the object's essence). That is, the specific location and rotation of an airplane in satellite imagery, or the 3d rotation of a chair in a natural image, or the rotation of a particle in a cryo-electron micrograph, do not change the intrinsic nature of those objects. Here, we consider the problem of learning semantic representations of objects that are invariant to pose and location in a fully unsupervised manner. We address shortcomings in previous approaches to this problem by introducing TARGET-VAE, a translation and rotation group-equivariant variational autoencoder framework. TARGET-VAE combines three core innovations: 1) a rotation and translation group-equivariant encoder architecture, 2) a structurally disentangled distribution over latent rotation, translation, and a rotation-translation-invariant semantic object representation, which are jointly inferred by the approximate inference network, and 3) a spatially equivariant generator network. In comprehensive experiments, we show that TARGET-VAE learns disentangled representations without supervision that significantly improve upon, and avoid the pathologies of, previous methods. When trained on images highly corrupted by rotation and translation, the semantic representations learned by TARGET-VAE are similar to those learned on consistently posed objects, dramatically improving clustering in the semantic latent space. Furthermore, TARGET-VAE is able to perform remarkably accurate unsupervised pose and location inference. We expect methods like TARGET-VAE will underpin future approaches for unsupervised object generation, pose prediction, and object detection. Our code is available at https://github.com/SMLC-NYSBC/TARGET-VAE.

Unsupervised Object Representation Learning using Translation and Rotation Group Equivariant VAE
Alireza Nasiri
Tristan Bepler
Simons Machine Learning Center New York Structural Biology Center anasiri @nysbc. org
Simons Machine Learning Center New York Structural Biology Center tbepler@nysbc.og
Abstract
In many imaging modalities, objects of interest occur in variety of locations and poses (i.e. are subject to translations and rotations in 2d or 3d), but the location and pose of an object does not change its semantics (i.e. the object's essence). That is, the specific location and rotation of an airplane in satellite imagery, or the 3d rotation of chair in natural image, or the rotation of particle in cryo-electron micrograph, do not change the intrinsic nature of those objects. Here, we consider the problem of learning semantic representations of objects that are invariant pose and location fully unsupervised manner. We address shortcomings previous approaches this problem by introducing TARGET-VAE, translation and rotation eooupeequivariant variational autoencoder framework. TARGET- VAE combines three core innovations: 1) rotation and translation group-equivariant encoder architecture, 2) a structurally disentangled distribution over latent rota- tion, translation, and which are jointly inferred by the approximate inference network, and 3) a spa- tially equivariant generator network. comprehensive experiments, we show that TARGET- VAF learns disentangled representations without supervision that signifi- cantly improve upon, and avoid the pathologies of, previous methods. When trained images highly corrupted by rotation and translation, the semantic representations learned by TARGET- VAE are similar to those learned on consistently posed ob- jects, dramatically improving clustering in the semantic latent space. Furthermore, TARGET-VAE is able to perform remarkably accurate unsupervised pose and location inference. We expect methods like TARGET VAE will underpin future ap- proaches for unsupervised object generation, pose prediction, and object detection. Our code is available at https: /tt///ggttttt/t com SMLCC NYSBC/TARGET VAE
semantic object representation,
Introduction
In many imaging modalities, objects of interest arbitrarily located and oriented within the image frame. Examples include airplanes in satellite images, galaxies astronomy images [ and particles in single-particks cryo-electron microscopy (cryo-EM) micrographs 2 However, neither the location nor the rotation of an object within an image frame changes the nature (i.e. the semantics) of the object itself. An airplane is airplane regardless of where it is in the image, and different rotations of a particle in a cryo-EM micrograph are still projections of the same protein. Hence, there is great interest in learning semantic representations of objects that invariant their locations and poses.
In general, unsupervised representation learning methods do not recover representations that disen- tangle the semantics of an object from its location or its pose. Popular unsupervised deep learning methods for images, such as variational autoencoders (VAE) [3), usually use encoder-decoder frame- works in which an encoder network is learned to map input image unstructured latent variable
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

or distribution over latent variables which is then decoded back to the input image by the decoder network. However, the unstructured nature of the latent variables means that they do not separate into any specific and interpretable sources of variation. To achieve disentanglement, methods have been proposed that encourage independence between latent variables 4 5 However, these methods make no prior or structural assumptions about what these latent variables should encode, even when some sources of variation e.g. an object's location or pose are prevalent and well-knowi Recently, several methods have proposed structured models that explicitly model rotation translation within their generative networks 6 7 g by formulating the image generative model as a function mapping coor- dinates in space to pixel values. Although promising, only the generative portion of these methods is equivariant to rotation and translation, and the inference networks have lackluster performance due to poor inductive bias for these structured latents.
To address this, we propose VARGET-VAE, a Translation and Rotation Group Equivariant Variational Auto-Encoder. TARGET-VAE is able to learn semantic object representations that are invariant to pose and location from images corrupted by these transformations, by structurally decomposing the image generative factors into semantic, rotation, and translation components. We perform approximate inference with a group equivariant convolutional neural network [ and specially formulated approximate posterior distribution that allows us to disentangle the latent variables into rotationally equivariant latent rotation, translationally equivariant latent translation, and rotation and translation invariant semantic latent variables. By combining this with spatial generator network, our framework is completely invariant to rotation and translation, unsupervised, and fully differentiable; the model is trained end-to using only observed images. In comprehensive experiments, show that TARGET- VAE accurately infers the rotation and translation of objects without supervision and learns high quality object representations even when training images heavily confounded by rotation and translation. We then show that this framework can be used to map continuous variation in 2D images of proteins collected with cryo-EM.
2 Related Work
In the recent years, there has been significant progress in machine learning methods for unsupervised semantic analysis of images using VAEs (6, 10 II [2). flow-based methods [13] Generative Adversarial Networks (GAN) 14 13 16), and capsule networks [17]. These methods generally seek to learn low-dimensional representation of each image in dataset, distribution over this latent, by learning to reconstruct the image from the latent variable. The latent variable, then, must capture variation between images in the dataset in order for them to be accurately reconstructed. These latents can then be used as features for downstream analysis, as they capture image content. However, these representations must capture all information needed reconstruct an image including common transformations that are not semantically meaningful. This often results in latent representations that group objects primarily by location, pose, or other nuisance variables, rather than type. One simple approach to address this is data augmentation. In this approach, additional images are created by applying a known set of nuisance transformations the original training dataset. A network is then trained to be invariant to these transformations by penalizing the difference between low-dimensional representations of the same image with different nuisance transforms applied 18 19. However, this approach is data inefficient and, more importantly, does not guarantee invariance to the underlying nuisance transformations. This general approach has improved with recent contrastive learning methods 20 21 22), but these ultimately suffer from the same underlying problems.
Other methods approach this problem by applying constraints on the unstructured latent space 15). In this general class of methods, the latent space is not explicitly structured to disentangle semantic latent variables from latents representing other transformations. Instead, the models need to be inspected post-hoc to determine if meaningfully separated concepts have been encoded in each latent variable and what those concepts are. A few works have addressed explicitly disentangling the latent space into structured and unstructured variables. Sun et al. [17 propose 3D point cloud representation of objects decomposed into capsules. Using a supervisdd learning scheme, their model learns transformation-nngg invarriant descriptors along with cewssformationeqquvariiii each capsule, improving reconstruction and clustering. Bepler et al. (6) propose spatial-VAE, a VAE framework that divides the latent variables into unstructured, rotation, and translation components. However, only the generative part of spatial- is equivariant to rotation and translation and the inference network struggles to perform meaningful inference on these transformations.
poses for

---

q
Encoder
q(0 (,,,))
Decoder
q(z (0.)))
g
Braken Facks
Inferecce modale
openn
Pixel values
Inforssoz
mobie
Pely boyss
Group Cuurdutionss
Fonne Maps
Figure I:
The TARGET-VAD framework. The encoder uses group-cquivarian convolutional lay- ers output mixture distributions over semantic representations, rotation, and translation. The cnnssformationeqqiiaan generator reconstructs the image based on the representation value z, and the transformed coordinates of the pixels.
To disentangle object semantic representations from spatial transformations, the inference network should also be equivariant to the spatial transformations. Translationeequivaiinn can be achieved by using convolutional layers in the inference model. However, convolutional layers are not rotation- equivariant. There has been many studies in designing rotation-ece eeuivirrinui neural networks
and specifically In Group-equivariant Convolutional Neural Networks (G-CNNs) [ 2d rotation space is discretized and filters are applied over this rotation dimension addition to spatial dimensions of the image. This creates network that structurally equivariant to the discrete rotation group the cost of additional compute. G-CNNs allow to create a rotation-equivarian inference network.
23 24 25]
A number of studies have proposed spatial transformation equivariant generative models, by modeling an image function that maps 2D or 3D spatial coordinates to the values the pixels those coordinates 7 26 27 6 Due this mapping, any transformation of the spatial coordinates produces exactly the same transformation of the generated image. However, other than Bepler et al. [6]. none of these studies perform inference on spatial transformations. To the best of our knowledge, TARGET- VAE the first method to have rotation and translation equivariance in both the inference and the generative networks to achieve disentangling.
3 Method
TARGET-VAE disentangles image latent variables into an unstructured semantic vector, and rotation and translation latents (Figure JI. The approximate posterior distribution is factorized into translation. rotation, and content components where the joint distribution over translation and discrete rotation groups is defined by attention map output by the equivariant encoder. The fine- fine-grained rotation and content vectors are drawn from a mixture model where each of the joint translation and discrete rotation combinations deviations of the continuous latents output by the encoder for each possible rotation and translation. Sampled translation, rotation, and content vectors are fed through the spatial decoder to generate images conditioned on the latent variables.
components of the mixture and used to select mean and standard
3.1 Image generation process
Images can be considered as the combination of discreetly identifiable pixels, and the spatial trans- formation an image is equivalent transforming the spatial coordinates of pixels. To have an image generation process adherent the transformations identified in the latent space, we define our

---

generator as function which maps the spatial coordinates of the pixels to their values. The generator outputs probability distribution as p(0: |2, Zy). where 2, I; and Di are the latent content vector, the spatial coordinate of the ith pixel, and the value of that pixel, respectively. Similar to B3 for an image with n pixels, we can define the probability of the image generated in this manner as sum over probability of its pixel values:
log p(i(2) = Σ log p(ú)),,
(I)
To define spatial coordinates over the pixels of the image, we use Cartesian coordinates with the origin at the center of the image. Translation and rotation of the image is achieved by shifting the origin and rotating coordinates around respectively. Depending on the value of the content vector, the generator acts as function over the pixel coordinates to produce an image. In this setup, the generation process is equivariant to transformations of the coordinate system. Assuming R(0) as the rotation matrix for angle 0, and translation value of t, the probability of the generated image from Formula
log p(0)z,,,1) = Σ log p(ù:lz, R(0);; +1)
In this study, we focus specifically on rotation and translation of the spatial coordinates, but the image generative process extends to any transformation the coordinate space.
3.2 Approximate inference on latent variables
We implement VAE framework to perform approximate inference on content, rotation, and transla- tion latent variables. Since rotation optimization is non-convex, we implement a mixture distribution over the 2D rotation space that allows the model to choose from discrete components. These components are used to approximate the posterior distributions over the rotation angle 0. We repre- sent the overall approximate posterior g(2, 8, t, rly), where 2 is the latent content vector, 0 is the rotation angle, and r refer to the translation and discretized rotation components, and is the input image. By making the simplifying assumption that q y) and q(l]t, y) are independent, the approximate posterior distribution factorizes as
q(2, 0, t, rly) q(2)t, r. (.l,,,y)440 ()) r) rly).
where rly) is the joint distribution over discrete translations and rotations and g(0 r, v) and ((2)6, r, y) are the distributions over the real-valued rotation and the latent content vector conditioned on r, and the input y.
The Kullback-Lebler (KL) divergence between the approximate posterior and the prior over the latent variables is then
4(2, r,,y),
Σ (22,0,6r/y)0og 2,0,1,
02,0,1.7)
To simplify this equation, we use the factorization from Equation to reduce the KL--divereence the following (see Appendix A1 for the full derivation), with the joint prior factorized into independent priors over the latent variables,
Klr + Eq(1,)) (KLg + KL.), where
(5)
KL,, Σ all, rly)rlg) ((t,rly)
p(t,r)
K KL (0 r, lp (0))). and
KL, KL (g (2)t, (x)).

---

Multiplication of Klg and L with g(t,r) in Equation weights the KL-divergence on A and by the joint posterior distribution over and We choose to make the priors, p(t) and p(r), independent giving p(t. r) = p())(r), but this is straightforward to relax if non-indepenndnt prior is desired.
We define the prior 0 to be independent of the translation and to be independent of the rotation and translation. We model 0 as being drawn from a mixture distribution where the mixture components are the discrete rotations, p(0)= = >. p p(()))((r). Given r. 0 is drawn from a Gaussian distribution with mean given by the angle offset defined by r and fixed standard deviation which depends on the number of discrete rotation groups and is set to x. We tighten each of the 9 mixture distributions by shrinking the standard deviation as r grows to reduce overlap between these distributions when the number of discrete rotations is large. For the prior on r, we usually use a uniform distribution, but a discretized normal distribution can also be used to bias the model towards certain rotation angles a non-uniform rotation distribution is known priori. Then, p(r) calculated based on foft for each discrete rotation. For example, when r = 4, then faet E {0. 2 T, 2 } which are the rotation angles of the discrete rotation group, P4. The prior over translation, Ł, is Gaussian with mean 0, which denotes the center of the 2D coordinate system. The standard deviation of this prior depends on the number of pixels in our fixed-size coordinate system and whether we want the model to be flexible with large translation values or not. Lower standard deviation for the prior over translations, will more heavily penalize large predicted translation values.
The full variational lower-bound for our model is
[log p(0)2, 0, t)]
(6)
(z,0,t)> ((=,0,Alv)
where the KL-divergence term is defined in Equation 5
3.3 Neural network architecture
3.3.1 Group convolutional encoder
Convolutional layers are translation equivariant, meaning that a spatial shift in the image causes the same shift in the output feature map, but are not rotation-equivaree equivariance in the inference model, we use group convolutional layers (.. symmetry group G' is defined as set X with an operation -, which is associative on the set, and has inverse and identity elements. Furthermore, combination of two symmetry transformations on a set, also creates symmetry group.
To incorporate rotation-
We define the P, group where N. and the operations consist of all translations, and discrete rotations, about any center of rotation in a 2D space. In the group convolutional layer, each kernel is rotated by 2 angles, where is E (0, 1.... r 1} and is then convolved with the input to produce an output map with r values corresponding to the rotations of the kernels.
Following the group convolutional layer, we apply three 1xl group convolutional layers (Figure 1}. which are efficiently implemented as 3D convolutions. The final layer outputs the parameters of (oftmax(a(y)))..
the approximate posterior distributions, qt (i,j), r' r'ly)
where a(y)
is an ræNaM activation map output by the network given input y. and g(2lt
A (u= oz (z and g(0) (i.j), r r',,) = -N(Howy)) + 2x 0o (v)) where Ms i',, oz (V) (. and oe (v))). are output by the inference network for each spatial location and discrete rotation group. r'2n is the angle of the r'th discrete rotation.
During training, we sample from the joint posterior distribution of q(t, rly) using Gumbel-Softmax 28 in order to differentiably sample from the approximate posterior to estimate the expected reconstruction error given g. We use the soft ooe-hot sample to calculate the weighted sum of the parameters of the q( ft r, y) and q(l)t, r, distributions and then sample from those distributions to get a sample from the full joint approximate posterior. We define a coordinate system where x is in the [-1,1] range, and is centered on the middle of the image. The output of the Guntel-Softmax directly deseribes a sample from the approximate posterior over the translation, which we get by calculating the weighted sum over the translation grid points.
5

---

3.3.2 Spatial decoder
Given and 0, we translate and rotate the image spatial coordinates, x' = R(0)x + t, where R(0) is the rotation matrix for angle 0. The generator receives the sampled = value along with transformed spatial coordinates, z', and outputs the value for the pixels at those spatial coordinates, forming the image. We use random Fourier feature expansion [29] on the coordinates of the pixel before feeding it through stack of fully-com connectdd layers (Figure TT. The spatial coordinate and inputs are processed separately by two parallel sets of fully-connected layers. Those representations are then summed and processed by the remaining shared fully-connected layers which give the final output value.
4 Results
4.1
Experiment setup
We run our experiments with P4. Ps. and PIe group convolutional layers in the inference model. The input images are normalized to the range of [0.1]. We use 128 rectangular kernels where size of the kernels in the first layer, is set to a value larger than the size of the targeted objects. The reason for group convolutional layer in our models and want assure that its
this is that only receptive field is at least as large as the objects of interest. We use leaky- ReLU activation functions, and the batch size for all the experiments is rate of 2e-4, and the learning rate is decayed by the factor of 0.5 after no improvements in the loss for 10 epochs. We run the training for maximum of 500 epochs with early-stopping in case of no improvements in the loss for 20 epochs. We run all our experiments in a single NVIDIA A100 GPU, with 80 GB memory.
100. We use ADAM optimizer [30], with learning
Each fully-connected layer in the generator has 512 hidden units and the generator output is a probability value, in the case of the binary valued pixels, mean and standard deviation for real valued pixels. For RGB images, the generator outputs R, G, and B values for each spatial location.
4.2 TARGET-VAE predicts the translation and rotation values of objects with high correlation
We first examine whether TARGET-VAE can accurately predict rotation and translation of objects despite being trained without supervision.
Datasets We use two variants of the MNIST dataset: 1) (MNIST(N)) MNIST digits randomly
rotated by sampling from A =²) and randomly translated by sampling from (0, 52) pixels, and 2) (MNIST(U) MNIST digits rotated by sampling uniformly from U(0, 2xr) and translated using the same distribution as in 1) (Appendix Figure 1). In both datasets, the train and test sets have 60,000, and 10,000 images of dimensions 50x50 pixels, respectively.
Training We train TARGET- VAE with P4, Ps. and Pie and z_ 2_dim=2 these generated datasets. For the prior over 0, we use (O +) for the MNIST(N) dataset, and U(0, 2T) for the MNIST(U). p(0), where fof are the rotation angles of the kernels in
For p(O(r) we calculate p(Ooff each dimension of For both datasets, we set the prior over / to (0,5²).
Given the trained model, perform inference We find the most likely values of L and 0 from the approximate posterior distributions given by the network and compare these with the ground truth values for the test set images. We calculate the Pearson correlation between the predicted and true translations and the circular correlation between our predicted and true rotation angles (3T] In comparison with a spatial-VAL network 66 trained with the same number of parameters as TARGET-VAE and z dim=2, we find that TARGET- VAE significantly improves over spatial- VAE when predicting rotation. Spatial- Spatial-VAE VAE fails to give meaningful rotation predictions when digits are uniformly rotated, but TARGET- VAE predicts rotation with remarkable accuracy (Table [). We also find that increasing the number of discrete rotations in the group convolution improves the accuracy of rotation inference, but does not have a significant impact on translation inference. Interestingly, spatial-VAL performs better than TARGET-VAE on translation inference, though the difference is small and predictions from both correlate strongly
and 0 on the test using the inference network.

---

with the ground truth. One possible reason for this is that TARGET-VA VAE only predicts whole pixel translations, due the formulation of the approximate posterior, whereas spatial spatial-VAE predicts real valued translations and, therefore, can predict sub-pixel values. In the future, including fine-grained translation inference in TARGET-VAF, similarly the rotation distribution, could allow TARGET- VAE to resolve sub-pixel translations well. We also found that increasing the dimension of did not have a noticeable effect on the ability of the network to predict translation and rotation.
Table I: Translation and rotation correlation on MNIST(N) and MNIST(U) with dimim2
MNIST(N)
MNIST(U)
Group Convolution
Translation
Rotation
Translation
Rotation
Spatial- VAB [6]
0.994, 0.993
0,564
0.982, 0.983
0.005
BARGET-VAE P4
0.97, 0.972
0.814
0.975, 0.976
0.80
TARGET-VAE Ps
0,981, 0.981
0.89
0,972, 0.971
0.859
TARGET-VAL P16
0.969, 0.972
0.898
0.974, 0.971
0.93
prdiissd
Figure 2: Left: Predicted 0 vs. ground-truth data in MNIST(N) Right: Predicted 0 vs. ground-truth data in MNIST(U), with TARGET-VAE Pg. There is k difference between the predicted angles and the ground-truth values of some of the digits (digits 0, and 8), which is related to the rotation symmetry them.
Some of the digits in MNIST have natural ambiguity in their rotations. Investigating the correlation between the predicted rotation values and the ground truth ones shows a difference about æ between some of the predicted angles and the ground truth (Figure 2}. This is expected, because several digits have approximate symmetry. For example, 0, 1, and are roughly 2-fold symmetric (Appendix A.3) creating inherent ambiguity in the rotation that is reflected in larger error in the rotation predictions (Appendix Table 1). Six and are also similar when rotated by म, although, interestingly, TARGET- VAE is able distinguish these digits and predict their rotations accurately. Perhaps this is because typical drawn with a straight spine while 6 is drawn with a curved one. This also suggests that the posterior predictions given by TARGET-VAL could be helpful for identifying the symmetry group of objects.
4.3 TARGET-VAE improves clustering accuracy by learning translation and rotation invariant features
By disentangling rotation and translation from the representation values, TARGET-VAE is able to recover semantic organization of objects. Using the z dim=2 models trained above and additional TARGET- VAE models trained with z dim=33,, we extract the most likely value of z for each model from q r. 3) for each image in the MNIST(N) and MNIST(U) test sets. To evaluate the disentangled semantic value of these representations, we cluster the test set images in z-space using agglomerative clustering [32]. We then calculate the clustering accuracy by finding the highest accuracy assignment between clusters and digits for each set of representations. We find that semantic representations produced by VARGET-VAE much more closely represent the underlying semantics of the MNIST dataset when compared to representations produced by baseline methods (Table 2). TARGET-VAE achieves clustering accuracies approximately 2-5x higher than standard VAE beta- VAE 4 and spatial-VAE (6) models trained with the same number of parameters TARGET-
7

---

Table 2: Clustering accuracy (%) on MNIST(N) and MNIST(U)
Model
MNIST(N)
MNIST(()
VAE (4_dim=4) 3
15.3
12.8
Beta- VAE (z_diinndd) dim=4)
15.1
18.0
Spatial-VAE (z_ dim=2)
66
37.1
28.2
TARGET-VAE P4 (z 2_dim=2)
56.4
56.6
TARGET-VAE P3 (2 (2_imm2)
60.1
57.1
TARGET-VAE P 16 dim=2)
60.1
63.4
TARGET-VAE 2 (7 v_dim=32)
65.1
64.3
TARGET-VAE P3 (z dim=32)
77.7
69.1
TARGET-VAE P16 (z_ dim=32)
75.2
71.2
VAE, We set the z_dim=2 for both spatial- and TARGET-VAE. Since the standard VAE and the bet do not have dedicated latent variables for identifying translation and rotation, we set the dimension of the unstructured latent space to 4 for these models. Also, we use = for the beta- VAE based the settings proposed by Higgins et al. [). We believe that direct inference rotation and translation with the equivariant inference network of TARGET-VAL VAE plus the transformation- equivariant generator are the main reason for the dramatic increase in the clustering accuracy (see ablation studies Appendix A.7).
We also observe that increasing the dimension of can significantly increase clustering performance and produces clearly distinct clusters for each digit (Figure B Increasing the number of discrete rotations also can improve clustering performance, likely because improving rotation inference means that z and are better disentangled, but this trend is not universally true for both MNIST(N) and MNIST(U). We note that the P& model with 2_dim=32 performs best on MNIST(N), perhaps because rotation inference is performed roughly equally well by the Ps and P16 models on this dataset.
1000
100
2
0
1
3
6
5
878
31
20
3
58
7
800
40
true Jabals
20
892
4
40
18
26
6
695
55
kn
131
-20
2
113
45
8
1
95
733
-40
7
37
183
12
26
17
661
24
14
40
31
8
158
1
2
3
4
5
6
7
predicted labels
Figure 3: Confusion matrix of trained TARGET-VAL VAE on MNIST(N) with z dim==2 (left), and the t-sne figure of the 32 dimension latent representation space (right). Based on the confusion matrix, most of the digits are clustered correctly, and the largest misclassification related to the confusion between rotated digits and 9, which is understandable.
TARGET- VAE is also able to detect multiple objects at inference time (Appendix A.4) and is able to learn meaningfully disentangled semantic representations on other datasets, including dSprites 33 (Appendix A.5) and the galaxy zoo [ (Appendix A.6).
4.4 Identifying class-averages and protein conformations in the cryo-EM micrographs with TARGET-VAE
In single-particle cryo-EM micrographs, individual biomolecules (particles) are spread throughout images with random orientations and locations. Being 2d projections of 3d structures, particle images have variability due to variation in the orientation of the particles sample and also have variability due to conformational heterogeneity between structures. To be able to identify the particles and

---

their conformations, semantic representations should be invariant to these transformations. Here, we demonstrate the cupacity of TARGET-VAE learning different classes and conformations of particles in cryo-EM micrographs.
Identifying different classes of particles in EMPIAR-10025 The dataset EMPIAR-10025 [34 contains T20S proteasome micrographs used to solve T20S proteasome structure at 2.8 A resolution. We the Topaz (35] to create a stack of 161,292 400x400 images of T20S proteasome particles. After downsampling by factor of 4 and normalizing the images, we train a Ps TARGET- VAE with dim=2. Due to the size and variability of these images, use larger spatial generator network than the one used on the MNIST datasets. Here, the generator has six fully-connected layers where each layer has 512 hidden units. We use uniform prior over r. We randomly select 10% of the images as the validation set to monitor the training process. After training, we apply the inference model the images of the particles extract the most- cesett-robblle semantic representations (same as the clustering of the MNIST datasets described in m.33). We find that these representations produce meaningful semantic clustering of the particles, grouping them by view and the presence of contaminants (gold particles in the dataset), and also can be used to reconstruct the particles with specified rotation and translation (Figure
Figure 4: Left: Images of T20S proteasome particles from EMPIAR- These images are extremely noisy. Right: The 2D latent representation space for EMPIAR-10025 learned by TARGET- VAE. TARGET-VAE identifies three clusters from which we show reconstructed examples produced by the spatial generator network.
Identifying different views of particles in EMPIAR-10029, EMPIAR- 10029 is a simulated EM dataset of GroEL particles. This dataset has 10,000 200x200 images. which we downsample to 100x100 pixels and normalize. We train the TARGET-VAE with the same settings as for EMPIAR- 10025. The latent space does not identify separate clusters in this dataset, likely because continuum over views is present from simulation, but exploration of the learned semantic latent space identifies GroEL projections independent of their rotation and translation (Figure
TARGET-VAE is also able discover continuous latent heterogeneity in other cryo-EM datasets (Appendix A.8).
5 Conclusion
We present TARGET-VAE, a translation and rotation group equivariant variational autoencoder framework for learning translation and rotation invariant object representation in images without supervision. By structuring the encoder to be translation and rotation equivariant, the model learns latent rotation and translation variables, disentangling these transformations from object semantics encoded in separately learned unstructured semantic latent variables, by training it jointly with spatially equivariant generator network. TARGET-VAB learns to accurately predict object locations and rotations, without any supervision, and also learns content representations that reflect known semantics (i.e., clusters match known semantic labels) across multiple datasets. Although we only consider 2d images containing single objects with rotation and translation, this framework can be extended to other object transformations by adopting encoder networks with the proper equivariances and adapting the approximate posterior distribution. In the future, expect this framework can also

---

Figure 5: Left: Simulated GroEL particle images from EMPIAR-10029, Right: particles. TARGET-VAE learns the distribution over particle views controlled for in plane rotation and translation and generates de-noised particle views.
The reconstructed
be extended to multi-object detection, object tracking over time, and 3d environments. TARGET- VAE lays the groundwork for a new generation of fully unsupervised object detection and semantic analysis methods for cryo-EM and other imaging modalities.
Acknowledgments and Disclosure of Funding
This work was supported by grant from the Simons Foundation (SF349247).
References
[1] Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Steven Bamford, Daniel Thomas, M Jordan Raddick, Robert C Nichol, Alex Szalay, Dan Andreescu, et al. Galaxy zoo: mor- phologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly 2008.
Notices of the Royal Astronomical Society,
[2] Fred J Sigworth. Principles of cryo-em single-particle image processing. Microscopy, 65(1): 57-67. 2016.
[3] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[4] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[5] Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint 03399, 2018.
[6] Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentangling image content from translation and rotation with spatial- VAE. Advances in Neural Information Processing Systems, 32, 2019.
[7] Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph Davis. Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature methods, 1862)_176-185. 2021.
[8] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference computer vision, pages 405-421. Springer, 2020.
10

---

[9] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999. PMLR, 2016
[10] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29, 2016.
[11] Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint arXiv: 1611 1.00013, 2016.
[12] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with con- volutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3412-3420, 2019.
[13] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
[14] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:/511.06433 2015.
[15] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation lcarning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[16] Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of contin- uous images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10753-10764, 2021.
[17] Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey E Hinton, and Kwang Moo Yi. Canonical capsules: Self-supervised capsules in canonical pose. Advances in Neural Information Processing Systems, 34:24993-25005. 2021.
[18) Luke Taylor and Geoff Nitschke. Improving deep learning with generic data augmentation. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1542-1547. IEEE, 2018.
[19] Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang. and Liang Lin. Clusternet: Deep hierarchical cluster network with rigorously invariatnnvariiant representation for point cloud analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1994-5002 2019.
[20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738. 2020.
[22] Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8392-8401, 2021.
[23] Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. Advances in Neural Information Processing Systems, 31, 2018.
[24] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint 17Xiv:1801.10130 2018.
[25] Erik J Bekkers. B-spline cnns on lie groups In International Conference on Learning Repre- sentations, 2019.
[26] Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of develop- ment. Genetic programming and evolvable machines, 132):131-122, 2007.
11

---

[27] Paul Andrei Bricman and Radu Tudor Ionescu. Coconet: A deep neural network for mapping pixel coordinates to color values. In International Conference on Neural Information Processing. pages 64-76. Springer, 2018.
[28] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1611.01144 2016.
[29] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in newral information processing systems, 20, 2007.
[30] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[31] S Rao Jammalamadaka and Ambar Sengupta. Topics in circular statistics, volume 5. world scientific, 2001.
[32] Fionn Murtagh and Pierre Legendre. Ward's hierarchical agglomerative clustering method: which algorithms implement ward's criterion? Journal of classification, 31(3):274-295, 2014.
[33] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset. https//ggthbb.comm//eppnn///psss datases,, 2017.
[34] Melody G Campbell, David Veesler, Anchi Cheng, Clinton S Potter, and Bridget Carragher. 2.8 à resolution reconstruction of the thermoplasma acidophilum 20s proteasome using cryo-clectron microscopy. Elife, 4:c06380, 2015.
[35] Tristan Bepler, Andrew Morin, Micah Rapp. Julia Brasch, Lawrence Shapiro, Alex J Noble, and Bonnie Berger. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nature methods, 16(11):1 (((1)):15331166 153-1160, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Our proposed framework can perform inference on the rotation and translation values of the object and use this inference learn rotation and translation invariant representation value for them.
(b) Did you describe the limitations of your work? [Yes] We mentioned the limitations in the conclusion in the form of the future directions.
(c) Did you discuss any potential negative socictal impacts of your work? [N/A] To the best of our knowledge, our work does not have any negative socictal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and we believe that our paper conforms to that.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In the parts that we made any assumptions, we clearly explained them.
(b) Did you include complete proofs of all theoretical results? [Yes] We included proof for one of our equations in section 3.2 which needs proof.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] The code will
be available after the double blind review process.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yss) All the details are either explained in the paper or can be found in the code.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] The variation in the results not large enough to justify using error bars.
12

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We included all the processing information.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r using/curating? [N/A] All the data that we are using is the public domain.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The data that any personally identifiable information or offensive content.
using does not contain
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "z4H9g3TH5fo", "reviewer": "Reviewer_Uca2", "summary": "The paper proposes TARGET-VAE, a representation learning method that disentangles the appearance, position, and rotation of an object in a given image. While the model is applied mainly to single-object images, it also provides some preliminary results that suggest that extending to multi-object images should be possible.\n\nThe method is basically a VAE with some inductive biases added to both encoder and the decoder. The inductive bias added to the encoder is what mainly distinguishes it from its main baseline -- Spatial-VAE. The central claim of this paper is that: using a group convolution network in the encoder leads to significant performance gains in disentangling the object rotation. Their empirical results show this to be true.\n\nThe encoder is basically a group convolution network -- which takes an image and applies a filter by both shifting and rotating the filter over the image. This gives a feature vector/cell for every possible shift and rotation. Each cell computes a logit which is used to select or activate the cell at a specific shift and rotation (using Gumbel-Softmax). The activated cell is then used to infer the latents for appearance, position, and rotation of the object. The decoder is directly taken from Spatial-VAE. It applies an explicit affine transformation to the object glimpse based on the given rotation and position.\n\n", "questions": "1. After doing Gumbel-Softmax in L180, how is the soft one-hot vector used? Is it used to weighted-sum the features of the G-feature map? Also, how is softmax applied on q(t,r|y) (in L223 and L245)? Is this followed by an arg max?\n\n2. The details of the prior applied on rotation $\\theta$ is unclear.\n\n3. In L180, is the q(t, r) a typo where the intention was to write q(t, r|y)?\n\n4. If my understanding is correct, then it might prevent confusion by saying $\\theta_{offset}$ 'belongs' rather than 'equal' to the set of angles in L137.\n\n5. Why not have a $\\Delta x$ for position similar to $\\Delta \\theta$ for rotation? It seems even the position can also suffer from discretization errors (just like rotation). Then, why not allow the position to also be freely adjusted/corrected using the selected $(t,r)$ feature. Also, another related question is: what is $x$ in L138? The confusion arises because (t, r, theta, z) are the only latents that are mentioned in equation (6).", "limitations": "Yes, the paper discusses the limitations and points to future directions such as possible extensions to other transformations (beyond rotation and position), multi-object scenes, and 3D scenes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel encoder design to help disentanglement of object rotation. \n2. Experiments focusing on single-object images show that the proposed model can infer rotation quite accurately while the baseline suffers significantly. The results also show that having a separate latent to represent the rotation relieves the burden on the appearance latent -- their results on clustering the appearance latent of MNIST digits show better clustering performance than the baselines.\n3. Paper does meaningful qualitative analysis of the model.\n4. Preliminary results suggest that extending to multi-object images should be possible. This may be of interest to the object-centric learning community where previous works have not explicitly decomposed the rotation angle. I feel perhaps this result could have been highlighted more in the main paper.\n5. Design seems extensible to some new factors of variation (not just position and rotation) by introducing an appropriate group convolution.\n\nWeaknesses:\n1. The disentanglement is still driven by the fact that the decoder performs an explicit rotation transformation of the canonical image by taking the rotation angle as input. This seems to suggest that when extending the model to other factors of variations such as color, one may need a decoder that takes the explicit color value as input -- this can become challenging very quickly when scaling to more such factors of variation.\n2. The model design and its presentation in the paper could be made cleaner. (see the Questions section for more details)\n3. It is a bit unjustified to say that the method 'detects and represents objects' without supervision because the model was trained on images that explicitly have one object. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "evrgd-3U_Aw", "reviewer": "Reviewer_Gmwc", "summary": "This paper presents an equivariant auto-encoder for object-centric representation. specifically, the group-equivariant encoder takes as input an image and outputs the disentangled latent representation -- an object is decomposed into pose and appearance. To learn the latent representation, the decoder reconstructs the input images in an equivariant way. In this case, the proposed method can learn the group-equivariant representation in an unsupervised style. Different from existing work, this method uses a better encoder and a better way to relate pose and appearance. The paper further validates their method in multiple datasets including a realistic application. And it shows better semantic learning and poses regression.", "questions": "Overall, my suggestion is to relate the work with Canonical capsules -- e.g., estimate the performance of canonicalization -- and include more details of multi-object experiments -- for example, we could analyze the detection performance, and canonicalization in the multi-object setting. ", "limitations": "The authors addressed the limitations well. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nThis work shows the potential of group equivariance in unsupervised representation learning, shedding light on symmetry-enforced learning. Different from previous works that show weak results, this paper gives a reasonable performance in both synthetic and realistic datasets. The only concern I have in this aspect is the missing discussion with closely related Canonical Capsules [1]. While the canonical capsule is proposed for 3D learning, it shares a similar target -- decomposing the object with disentangled factors. More importantly, both method shows the effectiveness of this decomposed representation, pose learning. It might be interesting to relate these two works. \n\n[1], Sun et al. Self-supervised capsules in canonical pose. NeurIPS 2021\n\nQuality:\nThe work validates their method with multiple datasets. The claims are well-supported. \n- One concern I had is the missing results in the commonly used benchmark for the unsupervised object discovery (the dataset used in Slot attention). In this case, we assume the objects are aligned. This might need to be justified. \n- And, to better understand the proposed method, one more suggestion I have is to perform the estimation of canonicalization as in the canonical capsule paper. When disentangling objects into pose and appearance, we already factor out the pose -- in some sense, we model the appearance of objects in the canonical pose. \n- I think multi-object experiments might be more interesting than single-object experiments. But the details of multi-object experiments are omitted. It would be cool to show more metrics like the performance of unsupervised detection. \n\nClarity:\nThe paper is overall easy to follow. From my viewpoint, the only concern is about the detail of the encoder. I think it might be more clear if the author can provide more implementation details regarding how to generate the different factors. \n\nSignificance:\nI think the results shown in this paper are interesting to our research community. It validates the effectiveness of group-equivariant representation in an unsupervised setting. In particular, the paper shows good results on multi-object images. This enables the group-equivariant to deal with the scene dataset. So I strongly recommend having more discussion in the multi-object experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PNeF5Tbakxu", "reviewer": "Reviewer_2xXA", "summary": "This paper proposes a model called Translation and Rotation Group Equivariant VAE (TARGET-VAE) which the authors claim can learn semantic object representations that are invariant to pose and location by structurally decomposing the image generative factors into the semantic, rotation, and translation components. Experiments are conducted on MNIST and a microscopy protein dataset to show that it can estimate the rotation and translation accurately.", "questions": "See above", "limitations": "Unfortunately, the authors did not include any limitations and potential negative societal impact discussion.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThe use of group equivariant convolution makes sense to me, and the results show that it can model rotation better than spatial VAE indeed. \n\nCons:\n\nThis paper is somewhat overclaiming and the experiment section is weak. The experiments are only conducted on MNIST and a protein dataset called EMPIAR-10025 with very few baselines. I'm not familiar with the protein dataset, but from the illustration in Fig.5, I don't think it could be called unsupervised \"object\" or \"semantic\" representation learning when only verifying on these simple datasets. If the authors genuinely believe their method is general and effective, I suggest they try it on more general RGB data like ImageNet, or at least CIFAR.\n\nFig.1 is unclear, making it hard to understand without referring to the text. I suggest the authors add more context to the figure and caption to make it more self-contained and help readers understand it better.\n\nThe motivation is not well addressed as the authors did not motivate why disentangling rotation and translation are essential for the generative models (maybe it is well known in the bioinformatics field but it is not reflected in this paper). The results also did not show the generated results but only the predicted translation/rotation results (I think a vanilla spatial transformer network should also be able to do that).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept