Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes an approach for pose estimation for objects with symmetries. The approach leverages SO(3)-equivariance to predict distributions over 3D rotations from single images. Image features are obtained from a pre-trained ResNet. These features are orthographically projected to the sphere where the features are convolved with a learned filter on S^2. As a result a signal is generated on SO(3). A group convolution on SO(3) is performed generating a detailed distribution over SO(3). This distribution can model symmetric objects. As a result object symmetries are learned and uncertainty can be represented. 
strength_and_weaknesses: The strengths include the ability to handle unknown object symmetries and the generation of distributions over poses, which in addition of allowing to take into account the pose ambiguity from object symmetries, it permits the representation of uncertainties due to partial observability. However with some datasets (SYMSOL) previous methods (Murphy et al.) have better performance. Computational cost not evaluated.
clarity,_quality,_novelty_and_reproducibility: The paper has good quality since it addresses the problem of object symmetries using equivariant layers and allows the representation of uncertainty. Some details are scarce, namely those concerning the computation of group convolution in Fourier space. The main contribution of the paper is the ability to provide uncertainty estimates for the poses of symmetric objects.
summary_of_the_review: This is a paper describing an approach for pose estimation of symmetric objects with prediction of uncertainty. The contribution is clear but incremental.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a deep neural network to predict poses in SO(3) from 2-D images. The proposed method first maps 2-D images onto a hemi-sphere by projection, and then convolve it with a filter on $S^2$ in the basis of spherical harmonics, and then apply rotation-equivariant layers that convolves the signals in the basis of Wigner D-matrices. The proposed method appears to be more effective than other networks on a variety of datasets in handling ambiguity that is caused by object symmetry.
strength_and_weaknesses: Strength: 

1. The paper aims to solve an interesting and challenging problem: estimating 3-D pose from 2-D images, which has broad applications in 3-D computer vision and 3-D imaging.

Weaknesses and questions:

1. Although the writing is in general ok, there are some missing details in the implementation. I encourage authors to provide a more detailed description of each layer and operations in the proposed neural network. For example, the authors did not clarify how exactly equation (2) is implemented in the Fourier space of SO(3).

2. The idea of this work is not sufficiently motivated. I can see the advantages of learning the distribution of SO(3) can mitigate the issue of symmetry. However, it remains unclear how the S2 convolution and the operations shown in Figure 2 are essential to the success of the proposed method. More explainations and experiments are expected to demonstrate the effectiveness of each module of the proposed network.

3. The idea of this work is not novel. It seems to me that proposed method is a combination of ideas from previous works such as group convolution,  Fourier expansion on SO(3) and etc. Learning distribution on SO(3) to break symmetry is also not a new idea, see [1].

4. The proposed method is not the only work that predicts rotations from 2-D images, so there are some missing references. See [2] that uses each 2-D image to predict the pose of protein molecule for 3-D reconstruction. I also encourage the authors to have some discussions that compare this work with your method, since they are able to obtain reliable pose estimates from 2-D images using a simple neural network.

[1] Synchronizing Probability Measures on Rotations via Optimal Transport.

[2] CryoAI: Amortized Inference of Poses for Ab Initio Reconstruction of 3D Molecular Volumes from Real Cryo-EM Images
clarity,_quality,_novelty_and_reproducibility: Novelty: As I commented, the idea seems not novel to me.

Clarity: More clarifications are needed to describe the proposed neural network.

Quality: The overall quality of this work is ok.
summary_of_the_review: In summary, this is an ok submission. However, there is a large room for improvement in terms of clarity and experiments. The idea of this work is neither novel nor sufficiently motivated.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Image2Sphere a rotation distribution estimation method by mapping 2D image features to the 3D rotation manifold and applying SO(3) equivariant convolution on the 3D rotation manifold. This is done by first mapping the image features onto a sphere using orthographic projection, and then a learned filter is applied to transform the signal onto SO(3). Finally, SO(3) group convolution is performed to produce the distribution over the 3D rotation space. Thanks to the SO(3) equivariant group convolution operation, the pose estimation distribution can be modeled well in the results. The method is evaluated on several image-based orientation estimation results and the proposed method is reported to outperform prior methods and other baselines, especially in a low data regime. 
strength_and_weaknesses: Strength:
1. The paper proposed a novel orientation distribution estimation method from simple image features. The idea of mapping image features to a sphere and further a 3D manifold is very interesting. 
2. SO(3)-equivariant Group convolution is devised for rotation estimation on the manifold. This brings a new idea to the field and I think it is inspiring and could be further extended to other applications. 
3. The paper is well-written and easy to follow.
 
Weakness: 
1. The evaluation is only conducted on images with a single object, well cropped and centered in the images.  I am wondering how the method performs in other more cluttered and occluded datasets (e.g. YCB-V and LM-O). Learned or ground truth detection/segmentation may be used for this purpose. 
2. I would like to hear the authors’ comments on how well the proposed method to unseen object instances/categories. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and easy to read. 

Quality: The equality is high, with a novel design and solid evaluation. 

Novelty: The proposed method is novel. 

Reproducibility: The source code is provided and there should no issues in reproducing. 

summary_of_the_review: The paper proposes a novel method for the distributional estimation of SO(3) rotation. The evaluation is solid and shows the efficacy of the method. I tend to accept this paper, yet the paper does not fully align with my background and I may not have a full understanding of the field.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a novel SO(3) equivariant architecture for pose prediction of objects from 2D images. The neural network takes image features from an image encoder and maps it to a sphere, which is followed by an $S^2$ convolution and one or more $SO(3)$ convolutions. Both these layers are equivariant to $SO(3)$ transformations. An important feature is that the output of the neural network is a distribution on the $SO(3)$ pose manifold. This feature is important in the case where the objects have symmetries and the pose can be known only up to that symmetry. Experiments on multiple pose prediction datasets show improved performance over baselines as well as the ability to produce distributions that show understanding of object symmetries.
strength_and_weaknesses: Strengths:

1. The proposed architecture and loss function -- I2S -- are simple and intuitive. 

2. The paper is well-written with good illustrations and visualizations of results for important cases that show the usefulness of the I2S and that I2S works predictably for some known objects, which is important.

3. Experiments on multiple datasets show that I2S indeed outperforms all the baselines considered almost always.

Weaknesses:

1. Novelty of the method: Given that both $SO(3)$-equivariant convolutions as well as pose prediction by predicting distributions over $SO(3)$ are existing ideas, I am not sure that the architecture in this paper is novel enough.

2. Comparison with Murphy (2021): Experiments on SYMSOL indicate that Murphy (2021) outperforms I2S when the entire dataset is available. However, I2S performs better in a few-shot setting. I think the authors should try to clarify why exactly mapping to the sphere and then the SO(3) is important, this seems to be the main difference from Murphy (2021), which is not clear in the current version of the paper. 

3. Experimental results should have error bars and statistical significance values.


clarity,_quality,_novelty_and_reproducibility: The paper is generally clear except what I wrote above and I believe that the experiments are reproducible in principle. I do think that the paper may not be novel enough for acceptance, but I hope the authors can convince me otherwise.
summary_of_the_review: Overall, I think the paper studies an important problem in computer vision, and proposes a simple and effective architecture, which takes into account symmetries in the objects for pose prediction. I do think some of the experimental results and novelty of the paper have to be explained and done better. I am inclining towards acceptance.

UPDATE AFTER AUTHOR RESPONSE:

I appreciate that the authors have responded too all my comments. I think the paper is in good shape to be accepted although architectural novelty is limited. It does show how to use existing spherical and SO(3) convolutional layers for an important problem and shows some improvements in performance. I am increasing my rating based on the authors' response to my comments and those of the other reviewers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies SO(3) rotation estimation from a single image and addresses symmetry and uncertainty by predicting a distribution in SO(3) with equivariant representation. State-of-the-art performance is presented on PASCAL 3D+. Although there are some clarity issues for the exposition and some complaint on novelty of architecture, all reviewers are positive about the work. 
justification_for_why_not_lower_score: The problem is important and the method is well acknowledged by all the reviewers.
Decision: Accept: notable-top-5%