Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel method (AQuaMaM) for 3D quaternion orientation estimation from potentially ambiguous 2D images. It employs a Transformer architecture to learn sequences of quaternion parameters representing distributions over the SO(3) group of 3D rotations, treating them as language tokens. 

The proposed architecture is able to efficiently learn multimodal distributions over SO(3), allowing to represent multiple legitimate candidate rotations corresponding to a given ambiguous 2D image.

The approach is validated on a toy dataset and a 3D die orientation estimation dataset, demonstrating higher accuracy and higher training and inference time efficiency with respect to a strong SoTA baseline (IPDF).
strength_and_weaknesses: Strengths
---------
- To my knowledge, the proposed approach is novel. It seems to be the first successful application of a Transformer autoregressive model for multimodal distribution estimation on SO(3).
- AQuaMaM is compared against a strong baseline (IPDF), demonstrating:
   1) Significantly faster convergence time
   2) Faster inference, since it requires few forward passes with respect to the $N$ required by IPDF
   3) Higher accuracy
   4) Higher reliability (see Fig. 6)
- The paper is generally well-written. The introduction is particularly pleasant to read and positions the paper very well in the literature.

Weaknesses
------------
- As detailed in the following, there are several points in which clarity shall be improved. This would especially help readers who may be familiar with the orientation estimation problem, yet not necessarily accustomed to language modeling and Transformers.
- The die experiment is convincing and allows for detailed analysis in a controlled way. Still, an additional comparison on a more challenging dataset (e.g., the one employed in the original IPDF paper) would strengthen the paper by validating AQuaMaM on an additional, possibly harder problem.
clarity,_quality,_novelty_and_reproducibility: Clarity
-----
- Writing is in general very good. Section 1 is particularly well-written
- Some statements and technical steps were not clear or simple to understand:
   1) "IPDF is trained with Ntrain ≪ Ntest which can make it difficult to reason about how the model will behave in the wild" --> I don't see how the number of employed test points and its relationship with the number of training points could impact performance expectations on unseen data.
   2) In Sec. 2.2, I found it hard to understand the role of the unit 3-ball. While reading, it was non-trivial to grasp why "Therefore, there is a one-to-one mapping f : B3 → H1". There seemed to be a gap w.r.t. the previous paragarph: it was not immediately obvious how the remaining quaternion parameters $q_x, q_y, q_z$ map to $B^3$
   3) In Eq. (1), the reasoning leading from the second expression (with the summation) and the third one is not clear. It goes from a summation of weighted uniform distributions to a single term weighted by $\pi_k$.
   4) After Eq. (2), the derivation of the conditional distribution term $p(q_z \| q_x, q_y)$ is not provided. It may possibly be trivial (or not), but in my view it should be at least reported explicitly in the Appendix.
   5) The origin of $q_w$ at the numerator in (4) is not clear. What is its relationship with $s_q$? I'd suggest to make the passages explicit.
- Some important points refer the reader to prior works. It would help a lot to provide a synthetic description of relevant tools in the main paper or possibly in the appendix to facilitate understanding. In particular:
   1) The partially causal attention mask
   2) The claim that "models using mixtures of uniform distributions that partition some domain can be optimized solely using a “language model loss”"
- Not clear whether inference requires a single pass as stated in the abstract or three passes (although optimized via caching) as mentioned at the end of Section 3.2
- In general, figures help a lot in grasping some of the most complex passages. Still, possible improvements may include: 
   1) Extending the scheme in Fig. 4 or adding a new one to illustrate the steps outlined in the last paragraph of Sec. 3.1, namely the generation of the transformed tokens and the classifier head; 
   2) The description below the underbrace in Fig. 4 may possibly be wrong. The first tokens should be patch embeddings, while the last 3 should be the position embeddings as far as I understood; 
   3) In Fig. 6b, the Category 4 subfigure should probably have 16 rotations, while only 3 are shown.
- There is some notational confusion and redundancy between $\pi_k, \pi_{q_c}, c, k$ and between $\omega_c$ and $\hat b_i$ from Figure 2
- Define $\pi_{q_{d,c}}$ and explain the difference from $\pi_{q_{c}}$
- Consider making the model parameters explicit in the NLL loss definition, i.e., $\mathcal{L}(\pi_{q_{d,c}}, \mathcal{X})$

Quality
----
- The quality of the work is generally high. The proposed architecture seems to be suitable for this kind of problem and is able to overcome some limitations of IPDF on well-designed experiments.

- Only one baseline is compared against (IPDF), but this choice appears justified by its strength with respect to other candidates (although on a different dataset).

- Training time to convergence is significantly faster for the proposed method w.r.t. IPDF, as shown in Fig.  5, supporting one of the main claims of this work 

-  I agree on the fact that the proposed die experimental scenario is more controllable and allows to better study some key aspects of the method and the baseline, as stated by the author in Sec. 4. Still, an additional comparison on the symmetric solids dataset employed in the original IPDF paper may be informative to evaluate the comparative performance of AQuaMaM on a more challenging task.

Novelty
---
- The approach appears indeed innovative. I am not aware of methods employing Transformer architectures to tackle this problem

Reproducibility
----
The approach seems to be described in sufficient detail for possibly reimplementing it. Implementation details and code snippets are reported in the appendix.
summary_of_the_review: Overall, in my view the paper makes a significant algorithmic contribution by presenting a novel Transformer-based approach to complex multi-modal distribution learning on SO(3). The proposed approach is demonstrated to be faster and more accurate and reliable than a strong baseline on two clear and controllable experimental setups.

The paper is generally well-written and pleasant to read. Some improvements would be required in terms of clarity, as detailed above.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper uses a quaternion model for capturing rotational distributions. The approach is branded as a new 'method' even if it appears more like a model construction. The efficiency over a baseline is demonstrated on simple data sets.

strength_and_weaknesses: *Strengths*

1. Quaternions are the standard way of modelling rotations in tracking, position estimation, and pose modelling, and thus most likely a sensible way of modelling them also for capturing complex rotational distributions in general.

2. The paper is interesting, and could topic-wise be a good fit for the conference.

*Weaknesses*

3. The originality and novelty of the approach is questionable: This appears more like a standard model design choice for the particular application (you know that the phenomenon has rotational symmetry, and thus you incorporate this as prior knowledge in your model), and not as a new 'method'.

4. The presentation could be improved. The text is wordy and could be stating things more directly and clearly. The abstract alone almost fills the first page.

5. The experiments act as proof of concept. Additional experiments would have strengthened the paper.

clarity,_quality,_novelty_and_reproducibility: The originality and novelty of the approach is questionable: This appears more like a standard model design choice for the particular application (you know that the phenomenon has rotational symmetry, and thus you incorporate this as prior knowledge in your model), and not as a new 'method'. The presentation could be improved. The text is wordy and could be stating things more directly and clearly. The paper has been written in first person ("I introduce...") which is non-standard and sounds a bit weird (this is a matter of taste, of course).

The author(s) have not shared code for replicating their experiments, but they provide pseudo-code in the appendix.

summary_of_the_review: This paper is interesting but appears to have multiple flaws.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes an approach to learning distributions on the SO(3) manifold via an autoregressive factorization of the distribution on quaternions, wherein each conditional and the first marginal are approximated by a mixture of uniform distributions.

The paper provides a comparison to the prior work of Murphy 2021, and shows some empirical advantages over that approach.
strength_and_weaknesses: Strengths:
* The method is relatively simple and very computationally efficient as compared to the approach of Murphy.
* The experimental results in figure 7 are illustrative.  This is a very nice choice of visual.

Weaknesses:
* The method is not very clearly motivated relative to other possibilities.  E.g. why use quaternions as the underlying representation on which to apply the autoregressive decomposition? And why use a mixture of uniforms rather than something else with a computable likelihood?
* The empirical component of the validation is very limited.  The authors could strength the paper with comparisons to additional methods.
* The experimental results in figure 7 A appear to be a negative result.  There is much probability mass spread throughout the space, not just on the four possible orientations given the top view of the “5”. 

clarity,_quality,_novelty_and_reproducibility: The choice to use a mixture of uniform distributions was surprising and a bit odd to me.  One would expect that problems like the ones examined would exhibit some amount of local smoothness that is not easily captured with the mixture of uniform construction.  Another option could have been, for example to use a mixture of Gaussians for each conditional (as, for example, in RNADE MOG https://www.pure.ed.ac.uk/ws/portalfiles/portal/11874029/1068.pdf).  Can the authors speak to this choice? Have they considered a mixture of Gaussians for the conditionals with appropriate rescaling (e.g. via softmax) to the constrained space?

A chief advantage of the approach is said to be the computational speed-up relative to Murphy 2021.  Is computation cost a limiting factor in the applications to which this method is applicable?   

The authors describe their method and characterize it relative to the work of Murphy 2021.  Why do the authors choose this as the base of comparison? I was not previously familiar with this approach so was surprised to see it treated as a gold-standard baseline to beat.  Without a compelling explanation for the choice of this baseline the empirical results do not stand on their own as impressive.

There has been quite a lot of recent work in this space.  Can the authors comment on advantages and disadvantages to previous work including:

De Bortoli, Valentin, et al. "Riemannian score-based generative modeling." arXiv preprint arXiv:2202.02763 (2022).

Falorsi, L., de Haan, P., Davidson, T. R., and Forré, P. (2019). Reparameterizing distributions on lie groups. In International Conference on Artificial Intelligence and Statistics, pages 3244–3253.

Leach, Adam, et al. "Denoising Diffusion Probabilistic Models on SO (3) for Rotational Alignment." ICLR 2022 Workshop on Geometrical and Topological Representation Learning. 2022.

---

The writing in the paper is generally very clear.
summary_of_the_review: The paper describes a new approach for modeling distributions on the SO3 manifold.  I recommend rejection because of little motivation for the choice of the method relative to other possibilities, and because of a lack of substantial positive empirical results to demonstrate that the method performs well.

Update: I have changed my score from 3 to 5.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: After discussions, the reviewers have converged on ratings 6 (KBHk), 5 (WNVz), 3 (oE5t), which would generally put the paper below the threshold of acceptance. Reviewers KBHk and WNVz engaged in extensive discussions with the author, but unfortunately oE5t did not engage. Personally I found the paper to be fairly well-written, as noted also by some of the reviews. Furthermore, the method enjoys fast inference, convergence, and accuracy. Although many details were discussed, the most significant issues are related to methodological novelty / originality and experimental validation.

Originality: I agree with oE5t and WNVz that the idea of applying an autoregressive model to the (discretized) coordinates of a quaternion is not super original. However, simplicity is also a virtue, and it could well be that this approach has advantages over earlier more complex methods. This would however need to be convincingly demonstrated.

Experiments: the experiments are limited to synthetic data with limited variability (e.g. just a rotated dice). I agree with the reviewers who noted that it would be nice to see experiments involving more challenging / real data. The vision, robotics, and autonomous driving communities have worked on pose estimation for a long time, and have various benchmark datasets and methods. Furthermore, to convincingly demonstrate the benefits of the presented method, a comparison to a wider variety of baselines would be preferred.

The paper only compares to IPDF, and as WNVz notes it is not clear that this should be considered the gold standard. The authors argue that they outperform IPDF and IPDF outperforms various baselines, so that direct comparisons to other baselines is not required. I do not agree with this line of reasoning though, because IPDF outperforms the baselines on a dataset they introduced (also synthetic), and the present paper outperforms IPDF on a newly introduced dataset (rotated dice), so transitive reasoning does not apply. A truly convincing demonstration of the performance of AQuaMaM would involve comparing a number of competitive and popular baseline methods on a number of real-world datasets.

My conclusion is that this is quite a promising work, and I do not agree that the paper is a "clear reject". However, the paper could be made much more convincing by systematically evaluating on a range of datasets, and directly comparing to a range of baseline methods evaluated under identical conditions. As such I encourage the author to continue to improve the work and to submit it to the next conference.
justification_for_why_not_higher_score: This is not a bad paper, but it could be much better if it did a proper empirical validation.
justification_for_why_not_lower_score: n/a
Decision: Reject