On the utility of Equivariance and Symmetry Breaking in Deep learning architectures on point clouds

Authors: Sharvaree Vadgama, Mohammad Mohaiminul Islam, Domas Buracas, Christian A Shewmake, Erik J Bekkers
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: deep learning architectures, geometric deep learning, equivariance, group convolutional networks, generative modeling

Abstract:
This paper explores the key factors that influence the performance of models working with point clouds, \edit{across different tasks of varying geometric complexity.} In this work, we explore the trade-offs between flexibility and weight-sharing introduced by equivariant layers, assessing when equivariance boosts or detracts from performance. It is often argued that providing more information as input improves a model's performance. However, if this additional information breaks certain properties, such as $SE(3)$ equivariance, does it remain beneficial?  We identify the key aspects of equivariant and non-equivariant architectures that drive success in different tasks by benchmarking them on segmentation, regression, and generation tasks across multiple datasets with increasing complexity. We observe a positive impact of equivariance, which becomes more pronounced with increasing task complexity, even when strict equivariance is not required.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730430318882

thread_type:
official_review

<thread_object>
creator: reviewer_5sub

<content>
summary:
This paper studies the impact of additional input information and SE(3)  equivariance on the performance of models processing point cloud data.  They present a series of hypotheses to  study different aspects of equivariant neural networks and symmetry breaking. Extensive experiments have been conducted on segmentation, regression, and generation tasks to verify the applicability and superiority of equivariant networks.

strengths:
1. Detailed introduction to previous work.
2. The relationship between task complexity and equivariance was studied, and it was found that the equivariant method has more obvious advantages in tasks that require strict equivariance. It is demonstrated that providing explicit geometric information can improve performance even in the case of symmetry breaking.

weaknesses:
1.There are so many contents from previous works that I can hardly tell the novelty. From my point of view, only Formulas 8 and 9 are new, but it is still quite easy to prove.

2.Although the paper proposes that symmetry breaking may be beneficial, it provides insufficient explanation of the mechanism behind it.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643552125

thread_type:
official_review

<thread_object>
creator: reviewer_hwht

<content>
summary:
This paper explores the key properties of deep learning networks on 3D point clouds, especially focusing on SE(3) equivariance. It conduct extensive experiments to test the initial hypothesis of the trade-offs between flexibility and weight-sharing introduced by equivariant layers. Based on the experimental results, a scalable network called Rapidash is introduced to facilitate the comprehensive testing of the hypothesis.

strengths:
1. Clear motivation and insightful key idea.   
2. The paper writing quality is very high, with explicit statements and clear logic links.   
3. The code is provided in the supplementary material.   
4. The analysis and theoretical conclusion may bring future insight to 3D point cloud learning.

weaknesses:
1. The major concern lies in the dataset. ShapeNet is a synthetic dataset. However, one important key challenge of modern 3D point cloud networks is their real-world performance on real-world data. Thus, whether the raised hypothesis can also be accepted in real-world 3D data needs to be explored.   
2. Although this paper verified the three important hypotheses, a **solution** derived from hypotheses such as some network design ideas or even *engineering tricks/techniques* (e.g., how to revise the current point convolution operation) should be further discussed and evaluated based on the extensive analysis of the paper, which will benefit the network design of future point cloud learning methods.  
3. The reviewer believes that some extra figure illustrations should be provided to better intuitively understand the proposed theoretical hypotheses.

*Conclusion*:   
The current version of the paper is theoretically important, but need more pilot studies to provide the **practical** values.

questions:
1. Some pilot studies (no need too much) should be conducted on real-world datasets, or at least some robustness point cloud datasets. For example, ScanObjectNN for point cloud classification, or ModelNet40-C for robustness point cloud recognition.
2. Some "solutions" should also be provided and evaluated (some pilot studies are enough).

If these two questions can be addressed well, the reviewer may consider raising the rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649927684

thread_type:
official_review

<thread_object>
creator: reviewer_wenf

<content>
summary:
Summary
This paper examines the role of equivariance and symmetry breaking in deep learning architectures for point clouds, focusing on the influence of additional input information and SE(3) equivariance on model performance. Through a series of experiments on various tasks and datasets (e.g., Shapenet 3D, QM9, and CMU Motion Capture), the authors compare the effects of equivariant and non-equivariant layers, exploring the advantages of equivariance as task complexity increases. Results show that equivariance offers significant benefits for small datasets and geometrically complex tasks, though this advantage diminishes in large-scale data regimes.

Strengths

Weaknesses

strengths:
Novel Research Problem: The paper presents several hypotheses regarding the impact of equivariant vs. non-equivariant architectures, particularly on tasks of varying complexity. This is a novel and relevant problem, providing new insights into geometric deep learning for point cloud data.
Comprehensive Experimental Design: The authors conduct extensive experiments across multiple datasets, thoroughly evaluating the performance of different architectures, thus providing strong support for the effectiveness of equivariance.
Modular and Scalable Architecture: The proposed Rapidash architecture is extensible, supporting various forms of equivariance, and provides an efficient platform for testing, which is beneficial for future research.
Clear and Substantiated Conclusions: The experiments confirm the advantages of equivariance in geometrically complex tasks and quantify the impact on different data scales, offering practical guidance for model selection and design.

weaknesses:
Limited Discussion on Symmetry Breaking: Although the paper addresses symmetry breaking by incorporating global coordinates, it lacks a detailed analysis of how this affects performance across different tasks. Additional theoretical insights or quantitative results would strengthen this discussion.
Insufficient Evaluation of Model Complexity and Computation Costs: While Rapidash shows promise for high-complexity tasks, the paper provides limited discussion on computational costs and scalability in practical applications, which may impact its usability for large-scale deployment.
Limited Benchmark Comparisons: While some baseline methods are included, the paper does not cover all recent benchmarks. Broader comparisons with state-of-the-art equivariant and non-equivariant methods would enhance the persuasiveness of the findings.
Lack of Practical Application Discussion: Although theoretically relevant, the paper does not discuss the potential impact of equivariance and symmetry breaking on real-world point cloud applications (e.g., 3D reconstruction or object detection), potentially limiting its practical value.

questions:
see Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699114697

thread_type:
official_review

<thread_object>
creator: reviewer_4why

<content>
summary:
This work investigates the utility of imposing equivariant constraints in models that perform point cloud processing. To study the effects of the application of different equivariant constraints, the authors propose a scalable architectural desing that allows for easily incorporating varying degrees of equivariance. They evaluate this architecture on different point cloud processing tasks, incorporating different levels of equivariance or symmetry-breaking factors that break the network's symmetry constraint. Experimental results are used to accept or reject a set of hypotheses regarding the effects of equivariant constraints on model performance and generalization. Specifically, this work analyzes the effects of equivariance when modifying the size of the model, the size of the dataset, the complexity of the task, or when allowing symmetry breaking by providing pose dependent information (that is not available in the typical equivariance case).

strengths:
- This work provides a detailed description of the most commonly used equivariant architectures for point-cloud processing.
- Additionally, the clear definition of the hypotheses that the authors aim to investigate allows for the explicit design of experiments that can be used to test them, facilitating the reader's understanding of the conclusions drawn from the experimental observations.
- The extensive experimental evaluation across diverse point-cloud tasks provides convincing evidence of the effect of difference levels of equivariance on the proposed Rapidash model, showing the influence of the dataset size and task complexity on these effects.

weaknesses:
- This work lacks sufficient attribution to prior work that has investigated hypotheses similar to the ones tested here. For example, hypothesis 5, on the effect of symmetry breaking in equivariant neural networks, has already been studied in prior works, such as :

	[1] Marc Finzi, Gregory Benton, Andrew Gordon Wilson "Residual Pathway Priors for Soft Equivariance Constraints"

	[2]  Mircea Petrache, Shubhendu Trivedi, "Approximation-Generalization Trade-offs under (Approximate) Group Equivariance"

	[3] Stefanos Pertigkiozoglou, Evangelos Chatzipantazis, Shubhendu Trivedi, Kostas Daniilidis , "Improving Equivariant Model Training via Constraint Relaxation"
- The experimental evaluation is limited to examining the effects of the equivariant constraints on a single architecture. While it is shown that these effects generalize across tasks it is not clear how they generalize across different model architectures. 
- In the experimental results, no error bars are provided. As a result, it is hard to assess the significance of the observations that are used to accept or reject the various hypotheses.

questions:
- Providing a more detailed discussion on how the proposed hypotheses have also been investigated in prior works would improve the completeness of this work.
- Why is it reasonable to expect that the experimental observations generalize across different model architectures?
- Including error bars and variances of the reported numbers, given the randomness of the model's initialization and training, would allow for easier verification of the significance of the reported observations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313444047

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal period

comment:
Dear reviewers and ACs

Thank you for your invaluable contributions to the review process. We are grateful for the time, effort, and feedback provided by the reviewers. 

We next address general points raised jointly among the reviewers and proceed to respond to specific comments made by reviewers. Firstly, we are glad that you found the problem at hand **novel problem** and found our approach **providing new insights** & **insightful key idea** (reviewer Wenf, reviewer HwhT). The proposed architecture in this work, Rapidash was found useful to conduct the experiments: **is extensible, supporting various forms of equivariance, and provides an efficient platform for testing, which is beneficial for future research** (reviewer Wenf). Our experimental analysis was well-received and reviewers found it **extensive experimental evaluation across diverse point-cloud tasks, extensive experiments across multiple datasets, thoroughly evaluating the performance** (reviewer 4wHY, Wenf ).  

The reviewers identify valuable points for improvement which we have incorporated in the updated version of the paper. The main additions and experiments in response to reviewers’ concerns include the following:

1. *Regarding lack of error bars as mentioned by reviewer 4wHY -* We have added variances to all the results presented in Table 1 and Table 2, as well as the best for the performing models in Table 3. ( We plan to add the rest of the results in Table 3 in the camera-ready version of the paper as they did not finish within the rebuttal pdf time period)
2. *Regarding the lack of attribution to related work (reviewer 4wHY) and limited discussion on symmetry breaking (reviewer Wenf, reviewer 4wHY),* we have added the suggested related work and more to the paper's main text. An additional paragraph discussing symmetry breaking is added in section 3.1 and Appendix 2.
3. *Limited benchmark comparisons (reviewer Wenf),* We have added additional benchmark results to Table 1, 2 and 3. 
4. *Lack of practical application discussion (reviewer Wenf, reviewer HwhT):* We have added additional experiments for classification on ModelNet40 and ModelNet40C datasets, as well as a discussion section discussing the findings based on the two datasets in the Appendix. 
5. *Clarity on contribution (reviewer 5SuB) and additional figure (reviewer HwhT)* we have updated our contributions in the main text and added an illustration of equivariant vs non-equivariant models on the segmentation task in the Appendix. 

We have addressed each of the reviewers' concerns and questions at length as a response to the reviews. We note that while we were unable to receive further feedback from 3 out of 4 reviewers regarding our implemented improvements, we are confident that the comprehensive additions and clarifications we have provided directly address the concerns raised in their initial reviews. We believe these substantial improvements strengthen the paper considerably and hope they will be taken into account in the final decision process.

Best
Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the utility of imposing equivariant constraints in models that perform point cloud processing. The manuscript was reviewed by four experts in the field. The recommendations are (2 x "5: marginally below the acceptance threshold", 2 x "6: marginally above the acceptance threshold"). The reviewers raised many concerns regarding the paper, e.g., limited technical novelty, unclear motivation and statement, unconvincing experimental evaluations, missing comparisons with previous literature, etc. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.