Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes TARGET-VAE, a representation learning method that disentangles the appearance, position, and rotation of an object in a given image. While the model is applied mainly to single-object images, it also provides some preliminary results that suggest that extending to multi-object images should be possible.

The method is basically a VAE with some inductive biases added to both encoder and the decoder. The inductive bias added to the encoder is what mainly distinguishes it from its main baseline -- Spatial-VAE. The central claim of this paper is that: using a group convolution network in the encoder leads to significant performance gains in disentangling the object rotation. Their empirical results show this to be true.

The encoder is basically a group convolution network -- which takes an image and applies a filter by both shifting and rotating the filter over the image. This gives a feature vector/cell for every possible shift and rotation. Each cell computes a logit which is used to select or activate the cell at a specific shift and rotation (using Gumbel-Softmax). The activated cell is then used to infer the latents for appearance, position, and rotation of the object. The decoder is directly taken from Spatial-VAE. It applies an explicit affine transformation to the object glimpse based on the given rotation and position.


strengths_and_weaknesses: Strengths:
1. The paper proposes a novel encoder design to help disentanglement of object rotation. 
2. Experiments focusing on single-object images show that the proposed model can infer rotation quite accurately while the baseline suffers significantly. The results also show that having a separate latent to represent the rotation relieves the burden on the appearance latent -- their results on clustering the appearance latent of MNIST digits show better clustering performance than the baselines.
3. Paper does meaningful qualitative analysis of the model.
4. Preliminary results suggest that extending to multi-object images should be possible. This may be of interest to the object-centric learning community where previous works have not explicitly decomposed the rotation angle. I feel perhaps this result could have been highlighted more in the main paper.
5. Design seems extensible to some new factors of variation (not just position and rotation) by introducing an appropriate group convolution.

Weaknesses:
1. The disentanglement is still driven by the fact that the decoder performs an explicit rotation transformation of the canonical image by taking the rotation angle as input. This seems to suggest that when extending the model to other factors of variations such as color, one may need a decoder that takes the explicit color value as input -- this can become challenging very quickly when scaling to more such factors of variation.
2. The model design and its presentation in the paper could be made cleaner. (see the Questions section for more details)
3. It is a bit unjustified to say that the method 'detects and represents objects' without supervision because the model was trained on images that explicitly have one object. 
questions: 1. After doing Gumbel-Softmax in L180, how is the soft one-hot vector used? Is it used to weighted-sum the features of the G-feature map? Also, how is softmax applied on q(t,r|y) (in L223 and L245)? Is this followed by an arg max?

2. The details of the prior applied on rotation $\theta$ is unclear.

3. In L180, is the q(t, r) a typo where the intention was to write q(t, r|y)?

4. If my understanding is correct, then it might prevent confusion by saying $\theta_{offset}$ 'belongs' rather than 'equal' to the set of angles in L137.

5. Why not have a $\Delta x$ for position similar to $\Delta \theta$ for rotation? It seems even the position can also suffer from discretization errors (just like rotation). Then, why not allow the position to also be freely adjusted/corrected using the selected $(t,r)$ feature. Also, another related question is: what is $x$ in L138? The confusion arises because (t, r, theta, z) are the only latents that are mentioned in equation (6).
limitations: Yes, the paper discusses the limitations and points to future directions such as possible extensions to other transformations (beyond rotation and position), multi-object scenes, and 3D scenes.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a model called Translation and Rotation Group Equivariant VAE (TARGET-VAE) which the authors claim can learn semantic object representations that are invariant to pose and location by structurally decomposing the image generative factors into the semantic, rotation, and translation components. Experiments are conducted on MNIST and a microscopy protein dataset to show that it can estimate the rotation and translation accurately.
strengths_and_weaknesses: Pros:

The use of group equivariant convolution makes sense to me, and the results show that it can model rotation better than spatial VAE indeed. 

Cons:

This paper is somewhat overclaiming and the experiment section is weak. The experiments are only conducted on MNIST and a protein dataset called EMPIAR-10025 with very few baselines. I'm not familiar with the protein dataset, but from the illustration in Fig.5, I don't think it could be called unsupervised "object" or "semantic" representation learning when only verifying on these simple datasets. If the authors genuinely believe their method is general and effective, I suggest they try it on more general RGB data like ImageNet, or at least CIFAR.

Fig.1 is unclear, making it hard to understand without referring to the text. I suggest the authors add more context to the figure and caption to make it more self-contained and help readers understand it better.

The motivation is not well addressed as the authors did not motivate why disentangling rotation and translation are essential for the generative models (maybe it is well known in the bioinformatics field but it is not reflected in this paper). The results also did not show the generated results but only the predicted translation/rotation results (I think a vanilla spatial transformer network should also be able to do that).
questions: See above
limitations: Unfortunately, the authors did not include any limitations and potential negative societal impact discussion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an equivariant auto-encoder for object-centric representation. specifically, the group-equivariant encoder takes as input an image and outputs the disentangled latent representation -- an object is decomposed into pose and appearance. To learn the latent representation, the decoder reconstructs the input images in an equivariant way. In this case, the proposed method can learn the group-equivariant representation in an unsupervised style. Different from existing work, this method uses a better encoder and a better way to relate pose and appearance. The paper further validates their method in multiple datasets including a realistic application. And it shows better semantic learning and poses regression.
strengths_and_weaknesses: Originality:
This work shows the potential of group equivariance in unsupervised representation learning, shedding light on symmetry-enforced learning. Different from previous works that show weak results, this paper gives a reasonable performance in both synthetic and realistic datasets. The only concern I have in this aspect is the missing discussion with closely related Canonical Capsules [1]. While the canonical capsule is proposed for 3D learning, it shares a similar target -- decomposing the object with disentangled factors. More importantly, both method shows the effectiveness of this decomposed representation, pose learning. It might be interesting to relate these two works. 

[1], Sun et al. Self-supervised capsules in canonical pose. NeurIPS 2021

Quality:
The work validates their method with multiple datasets. The claims are well-supported. 
- One concern I had is the missing results in the commonly used benchmark for the unsupervised object discovery (the dataset used in Slot attention). In this case, we assume the objects are aligned. This might need to be justified. 
- And, to better understand the proposed method, one more suggestion I have is to perform the estimation of canonicalization as in the canonical capsule paper. When disentangling objects into pose and appearance, we already factor out the pose -- in some sense, we model the appearance of objects in the canonical pose. 
- I think multi-object experiments might be more interesting than single-object experiments. But the details of multi-object experiments are omitted. It would be cool to show more metrics like the performance of unsupervised detection. 

Clarity:
The paper is overall easy to follow. From my viewpoint, the only concern is about the detail of the encoder. I think it might be more clear if the author can provide more implementation details regarding how to generate the different factors. 

Significance:
I think the results shown in this paper are interesting to our research community. It validates the effectiveness of group-equivariant representation in an unsupervised setting. In particular, the paper shows good results on multi-object images. This enables the group-equivariant to deal with the scene dataset. So I strongly recommend having more discussion in the multi-object experiments. 
questions: Overall, my suggestion is to relate the work with Canonical capsules -- e.g., estimate the performance of canonicalization -- and include more details of multi-object experiments -- for example, we could analyze the detection performance, and canonicalization in the multi-object setting. 
limitations: The authors addressed the limitations well.
review_title: null

Meta-review: metareview: The authors propose an effective method for group-equivariant representation learning in an unsupervised setting. The authors use group convolutional encoder in the VAE setting. This is an interesting problem in the community. The proposed method makes sense and seems technically sound. The experiment results are good and convincing in general. As pointed by multiple reviewers, however it would have been better to focus more on multi-object settings. Most of the concerns raised by the reviewers are well addressed. Although reviewer 2xXA gave relatively lower score 4, the reviewer admitted his/her lack of related background knowledge and the confidence score was low.
confidence: Certain
Decision: Accept