Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the author explores the direction of how to utilize SE(3)-equivariance as a novel inductive bias to facilitate sample efficiency. Specifically, it focuses on the same robotic manipulation task as previous work, Neural Descriptor Field. The authors derive an energy function based on the representation of the Lie group, which can be used to characterize the distance/discrepancy between SE(3) pose in the demonstration and the current state. By approximating the gradient of the function with sampling, the SE(3)-equivariance demonstration following problems are formulated as an energy minimization problem. Experiments on the object placement task show that the proposed method outperforms the modified-Transporter Network and Invariant Descriptor Fields method. 

strength_and_weaknesses: This paper is trying to solve a very important problem: how to encode the SE-(3) equivariance property into the network as a powerful inductive bias. It is an important technique for many 3D vision and robotics applications. Although the TransporterNetwork shows that such kind of network design is helpful for manipulation tasks, it still relies on the translational invariance of CNN but does not explore this problem in a more general SE(3) space and does not deal with the hard rotation part. The authors leverage the representation theory of third-order Lie groups to define the bi-equivariant energy using the probability distribution on SE(3). The formulated equivariant descriptor field can then be used in a key-query system, as previous NDF. I feel that the energy-based model in 4.2 makes much sense. But the implementation choice of using Dirac delta functions to shape the query density makes this design not so useful in many applications where continuous parameterization is more preferred.

One weakness of this paper is the gradient of EBM, which requires a special sampling strategy described in Section 5. It seems that this design makes the overall complexity of the method very high in applications and leads to a very long inference time. As the author discussed in the final section, this method cannot solve problems at the trajectory level. 
clarity,_quality,_novelty_and_reproducibility: For the paper writing, the relationship between the bi-equivariant EBM and the bi-equivariant energy function is not so clear to me. For novelty, many previous works also explored the same SE(3)-equivariance, and many formulations are similar to previous work, e.g. NDF. The code is attached with well-written documentation, so I believe that it is easy to reproduce. 
summary_of_the_review: I feel that the problem solved in this paper is well-motivated, and the problem formulation looks good to me. However, the implementation choice can be improved to better support the theoretical power of EBM. The difficulty level of the experiments is on-par with previous work in this field. So I recommend a weak acceptance for this paper but the rating may be lower or higher after the rebuttal phase.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents SE(3) equivariant energy-based models for sample-efficient visual robotic manipulation from point clouds. It also provides theoretical conditions for bi-equivariant energy-based models, along with sampling strategies over the SE(3) manifold. Experiments on simple simulated pick-place and stick-in-tray tasks suggest the feasibility of  generalization to unseen target poses and unseen target object instances of the same category.
strength_and_weaknesses: - The paper deals with an important problem in robot manipulation. Learning equivariant and background invariant policies is key to visual robotic manipulation. Some recent work has addressed this problem empirically, but formal treatment of the problem has been widely neglected.

- The work builds upon Neural Descriptor Fields (NDFs) work in Simeonov et al., 2021. The authors claim that point cloud input needs to be segmented from the background in NDFs, and requires pre-training on an excessive dataset of objects. But do not validate these claims with grasping in clutter and/or multiple objects or challenging objects.

- Comparison with Dense object descriptors and grasping methods for visual robotic planning methods would further improve the quality of the paper. At this point, it is hard to comment on the efficacy of the approach in comparison to the state-of-the-art in visual robotic manipulation.

- Grasping in clutter and/or challenging objects would further develop the limitation of the presented sampling methods.

clarity,_quality,_novelty_and_reproducibility: The paper is very densely written, and formalizes the problem with representation theory of the Lie group. A lot of necessary implementation and experimentation details are in the appendix only. It would be helpful to state the key result and ground the claims of the paper with more experimentation.
summary_of_the_review: SE(3) equivariant energy-based models are an important direction of research for robot manipulation. The paper adequately formalizes the problem. Initial results are promising, and would need more experiments to ground the claims of the paper and establish its usability for visual robotic manipulation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a SE3-equivariant EBM with high sample efficiency and a competent generalization capability. 
strength_and_weaknesses: Strength
* The proposed method is well-motivated theoretically, and its practical instantiation incorporatoins several components to accomendate the innate instability and intractability in the theoretical formulation. 
* The paper offers an analysis on the generalization capability and robustness of the proposed method under different unseen conditions. 

Weaknesses
* The main concern is that the empirical verification is very limited. The tasks being evaluated are restricted to one bug-hanging task and one stick-to-tray task. It is unclear whether the proposed method will encounter practical problems in general.
* It will be helpful if the paper can provide an analysis on how much demonstration is needed for the method to succeed, and how the learning curve saturates with respect to the amount of demonstration data available.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The proposed theoretical formulation is novel. 
summary_of_the_review: This work introduces a SE3 equivariant formulation for robot manipulations that is fully differentiable, and empirically verifies the sample efficiency, generalization capability and robustness of the proposed method. The tasks being evaluated, however, are highly restricted. The work has a nice theoretical guidance and more empirical evidence that it works well in practice would improve the contribution of this work. 

--- Post Rebuttal ---

I believe the formulation of this work can inspire further research, and it has empirical support following the evaluation protocol from the literature. I've raised my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an end-to-end visual learning method for 3D robotic manipulation. The authors construct an SE(3)-equivariant energy-based model to learn end-to-end from limited demonstrations without prior knowledge. The proposed Equivariant Descriptor Fields could be generalized to unseen poses, instances, and target objects.
strength_and_weaknesses: This paper proposes a data-efficient model that could learn end-to-end for robotic manipulation from visual demonstrations. Compared with previous SE(3) - equivariant models, the proposed method could learn from only a few demonstrations without pre-training. The authors have proposed an interesting solution to solve the sample efficiency problem, yet there are still concerns about the experiment part to demonstrate the proposed method. Please refer to the below comments for my questions about this paper.

Method:
1. According to the paper, the major contribution is that the authors propose an EBM to model the descriptor field. However, the related content only takes a small part, and most materials in section 2 are about the description of SE(3)-equivalent. The author should emphasize more on the proposed contribution, while the less relevant concept and formulas could be left in citations.

2. For the proposed EDF, there are related works about MCMC sampling [1][2] and relevant learning methods for point clouds, such as [3]. It is essential to introduce and compare these related works since the EDF is the major contribution in this paper.

Except for the proposed method, I also have some concerns about the experiments.

1. The author has only shown the experiment in the task of mug hanging. However, as the author has mentioned about Neural Descriptor Fields (NDF), there are additional manipulation targets used in the simulated environment, i.e., bowl and bottle. Why not the author conduct these experiments to show the generalization of the proposed method?

2. Although the author has claimed that the model could learn without prior segmentation from the background, the experiment is conducted in a very simple environment. This raises concerns if the EDFs (or IDFs) could perform as expected when the environment becomes complicated, such as introducing random distracting objects.

3. Although the author has mentioned that NDF requires pre-training, it is self-supersized in an auto-encoding way that requires no expert labels. Therefore it is of great interest to see the comparison between NDF and the proposed method since both models could learn from a few demonstrations. This should be listed in the experiment section.

4. In the first paragraph in section 6, the author should mention the computation architecture when mentioning the processing time. Additionally, since the author has claimed the learning efficiency of NDF, it is worthwhile to compare the time efficiency in the experiments.


Writing:
1. In the experiment section, the author mix uses different tenses in the experiment settings and descriptions (also found in other sections). 
2. In the last paragraph of section 3, Transporter networks ->  Transporter Networks
3. Background related to EBM is missing in section 2.

[1] Zhu, Song Chun, and David Mumford. "Grade: Gibbs reaction and diffusion equations." Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271). IEEE, 1998.

[2] Xie, Jianwen, et al. "A theory of generative convnet." International Conference on Machine Learning. PMLR, 2016.

[3] Xie, Jianwen, et al. "Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

clarity,_quality,_novelty_and_reproducibility: Clarity: The author has listed too many less relevant materials on basic concepts about the Group Theory and misses emphasizing its major contributions about the EBM part.

Quality, Novelty: In my view, the major contribution is the author proposed an EBM for the SE(3)-equivalent field descriptor. However, there lacks evidence in the experiments to demonstrate the advantage claimed by the author.

Reproducibility: The author has claimed to submit all the codes to reproduce the results of all the experiments in the paper. If this could be revealed it would be convenient for other researchers to follow the work.
summary_of_the_review: This paper has proposed an EBM to learn end-to-end 3D robotic manipulation from limited visual demonstrations. The major concern from me about the effectiveness of the proposed method is the experiment section, where the author mainly conducts the experiment in a simulation environment and merely in the pick&place task of a mug. Since the author has claimed no pre-segmented points cloud info is required in the proposed method, the author needs to show the generalization on different manipulation objects. Furthermore, in the compared methods, i.e., Transporter Networks (TN) or Neural Descriptor Fields (NDF), the authors conduct the experiments in multiple situations to demonstrate the effectiveness of their proposed methods, such as 10 different manipulation tasks in TN and real-world experiment in NDF. Please refer to the previous comments for my concerns.

---Post Rebuttal----

First, I'd like to move my rating from 3->5 due to the significant revision of the paper and the updated experiments. However, as I am not an expert in the field of robotics, I still have concerns about the experiment part. Although the author has updated the experiments with more objects, I am not sure about the meaning of the comparison result. Are there any other metrics other than success rate and computation time to compare different models? Are they any other methods to compare in the pick-and-place task except for SE(3)-TN?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed Equivariant Descriptor Fields (EDF), a generalized, incredibly low-data-needed, fully end-to-end-training model which is SE(3)-equivariant (this means apply 3D translation and rotation to input will results to the same as it is applied to the output).

Authors utilized the representation theory of Lie groups and designed a novel SE(3)-equivariant energy-based model. It has high sample efficiency. It can also be learned from scratch using a few data (5-10 demonstrations). The EBM used MCMC-sample-based MLE to train.

They experiment EDF on 6-DoF robotic manipulation tasks (pick-and-place cups and etc.). Compared with TN (an SE(2)-equivariant method) and IDF (an SE(3)-invariant method, similar to NDF), EDF showed its sample efficiency and generalizability.
strength_and_weaknesses: - Strength
  - The paper is well written, at least non-math part. 
  - The problem formulation is good and the designed method seems technically sound.
- Weaknesses
  - As an non-math background reader, I cannot find supportive discussion on why this new approach can achieve such advantages (very few training examples and good generalization method). 
  - Even though it is normal practice as a theoretical paper, the experiment is too naive.
    - Only two tasks are tested, it is hard to prove anything.
    - With that number of data (5-10) and smaller than 5% difference, I do not think any conclusion can be made. The author should at least do the same experiments multiple times to provide confidential interval for accuracy.
    - You should provide more background research related to EBM, especially EBM on controlling. Such as [1] and its following paper; [2].

--- after rebuttal --- 
readability improved. Experiments added. Background added. Problem addressed. I raise the score.

- Question:
  - Time cost is not provided but EBM must be incredibly slow. Is it correct? Is efficiency a bottleneck for scalability? Both in training and applying.
  - IDF is SE(3)-invar. , NDF is SE(3)-equivar. Why IDC can  be understood as an end-to-end trainable version? Appendix E do not have a clear explanation.

[1] Xie, Jianwen, et al. "A theory of generative convnet." International Conference on Machine Learning. PMLR, 2016.
[2] Y. Xu, et al.  "Energy-Based Continuous Inverse Optimal Control," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109/TNNLS.2022.3168795.
clarity,_quality,_novelty_and_reproducibility: - Clarity: just-so-so
  - Not quite friendly for people of non-mathematic background. Why Lie-group? Why it is bi-equivariant? etc.
  - Non-math part is clear.
- Quality: needs improvement
  - No supportive discussions;
  - Too naive experiments.
Novelty: good
  - If the advantage is well proved in mathematical proof or well-defined experiment, it is important and useful for not only robot manipulation, but many domains.
  - Designs to solve this problem are novel and technically sound.
- Reproducibility
  - Code submitted and said reproducible but not checked.
summary_of_the_review: This paper designed a good method to solve an important problem. Multiple advantages are presented but not validated. It lacks supportive discussions and experiments.

Since I do not have a very strong math background, I am not able to access the correctness and novelty of this paper from the perspective of math. If a math guy evaluated this paper as perfect to be accepted. Just ignore my weakness assessment.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper uses the representation theory of the Lie group to construct a novel SE(3)-equivariant energy-based model for improving sample efficiency in learning robotic manipulation. The proposed method can learn the manipulation from a few visual demonstrations without pre-training.  Experiments are conducted to verify the effectiveness of the models, including sample efficiency and generalizability. The paper studies an important problem in learning for robotic manipulation, both theoretical analysis and experimental results are sufficient to justify the idea and demonstrate the performance. During the rebuttal, the authors improved the paper by adding extra supportive experiments and revising the paper according to the suggestions provided by the reviewers. Major concerns have been addressed and all five reviewers lean to accept the paper, with one championing it. After reading all the comments and rebuttals and having an internal discussion with the reviewers, the AC agrees with the reviewers and thinks that the current paper presents a solid research work that is worthy for publication, thus recommending an acceptance. The AC urges the authors to improve their final paper by taking into account all the suggestions from the reviewers. 

justification_for_why_not_higher_score: The main drawback of the paper is the lack of justification in many real robot applications, but it doesn't affect the acceptance decision. 
justification_for_why_not_lower_score: All five reviewers agree on accepting the paper because of the novel idea, sufficient empirical results, and valuable theoretical contribution in the paper.
Decision: Accept: poster