Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work proposes a Fully Convolutional Network (FCN) for learning dynamics of granular object piles. The key intuition is that, unlike alternative modeling choices (fully connected, graph, etc.), the inductive biases of FCN match the local nature of the granular dynamics. States and actions are rendered into grids and the resulting dynamics are utilized for curvilinear trajectory optimization. The work provides theoretical and experimental results validating that this representation is well fitted to the problem and outperforms alternative methods at granular object sweeping, including with obstacles.
review: Strengths:
- Main contribution is simple and fits to the task at hand.
- Experiments are thorough, comparing to several state of the art methods and demonstrating clear performance gains.

Weaknesses:
- Setup assumes full observability of granular material and obstacles.
- Analysis is missing some clarity in explanation.

The FCN contribution builds on well understood principles and applies it effectively to the granular object manipulation task, and in a novel way. One form of related work that should be added is Reinforcement Learning based methods (e.g., "Learning to Manipulate Object Collections Using Grounded State Representations," Matthew Wilson and Tucker Hermans, CoRL, 2020). 

Another work to be aware of is "Dynamic-Resolution Model Learning for Object Pile Manipulation" by Yixuan Wang, Yunzhu Li, Katherine Driggs-Campbell, Li Fei-Fei, and Jiajun Wu, RSS 2023, which proposes using predicted resolutions of particles to avoid inefficiency in GNN-based dynamics and demonstrated impressive real world results. If possible, this work should be compared to, as this is (to the reviewer's knowledge) is the current SOTA in GNN-based granular dynamics.

The theoretical analysis performed in Sec. 3.2 is difficult to follow. In particular, it is not clear how to related Eqs. 1 and 2 and the derivation of the correlation term following Eq. 2 is also unclear. Further explanation of why these terms are important along with their derivations (potentially in appendix) would strengthen quality of result.

The experiments demonstrate significant results, including generalization to new environments, good performance scaling, and importantly the ability to reason over obstacles and multiple task objectives with a single method. The performance is encouraging for the use of learned dynamics trajectory optimization in granular material manipulation.

Other notes:
- Sec 3.2 line 137 says Connected instead of Convolutional, which is important to be clear about.
- Appendix B.1, line 441 I believe should refer to Fig. 3.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The proposed solutions is simple and effective and the experiments are thorough and show strong performance compared to baselines. I am unsure of the technical soundness of some of the theoretical analysis surrounding the inductive biases and have asked for clarification; I would want to check that more thoroughly before accepting. Additionally, I have asked for comparison to a recently released work from RSS 2023 that updates the SOTA in GNN-based dynamics which would provide a more up to date baseline for the method. Assuming good performance there, I think the paper presents a strong case.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a learning-based method for granular material manipulation. The scene is represented by voxels, and a fully convolutional neural network operates on a density-filed-based representation for the object piles. The dynamics network learns the state transition and another differentiable rendering module can map the state into the trajectory. The entire pipeline can be used to perform trajectory optimization.
review: Strength

The authors make a real-world implementation and the results shown in the supplementary video look impressive. 

This pipeline is end-to-end differentiable so can utilize the gradient-based optimization for planning, which is efficient and powerful.

In the experiments, the proposed method has decent performance and outperforms other baselines. It can also achieve reasonable results in versatile environments.

Weakness

The dynamics networks lack interpretability. The correctness of the learned state transition isThe real-world experiment is great. I’m curious how much effort is used in sim-to-real transferring. Moreover, what is the success rate? Can the robot arm consistently achieve the goals? hard to verify.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The idea of this paper is clear and intuitive. It is nice to learn a fully differentiable simulation and rendering pipeline and further use the learned model to perform gradient-based optimization. The experiments are convincing that the methods can be implemented on real robot arms, stay robust under different settings, and outperform baseline methods. Therefore, I recommend acceptance.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes a method that uses a Fully Convolutional Network (FCN) to encode granular dynamics, also it utilizes differential render to construct loss for dynamics and trajectory optimization. This method is efficient compared with particle-based methods and it demonstrates good performance for simulation and real-world tasks. 


review: Strengths:

- Use FCN to capture granular dynamics, and enable a fixed-resolution approach for efficiency.
- Use curvilinear trajectory optimization for obstacle avoidance during manipulation.

Weaknesses:

- Fixed resolution FCN may lose some details for smaller granular.

I have a question about the density field representation, does there exist a trade-off between field resolution and dynamic modeling accuracy? Can FCN model small particles well?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this is an interesting paper that can be accepted. Overall the experiments show the effectiveness of the method and the framework is sound.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces Neural Field Dynamics Model which uses a network to obtain a density representation of piles of granular objects. The action rendering module also proposed in this work is fully differentiable and can be used for learning curvilinear trajectories, rather than only linear trajectories, which is a limitation of prior work. Curvilinear trajectories advance this work and allow it to perform obstacle avoidance when moving granular objects to target locations. 
review: The obstacle avoidance with moving granular objects is a novel problem introduced in this paper. 

The paper is technically sound, with clear explanations for trajectory optimization with learned dynamics. When introducing the model, reference the appendix for model details. Otherwise, at a first glance of the paper, it seems like there are no model details. 

Trajectory optimization is very clear and well explained but the correlation equation is hard to understand as there is little explanation. Results and experiments are significant and well explained. 

The limitations of the work are well explained and I did not find many other drawbacks on the work. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I give this paper a weak accept over a strong accept due to lack of clarity in the discussion of how correlation is calculated, little explanation for the result shown in Figure 3, and the fact that the baselines were hard to understand. It is an accept over a reject as trajectory optimization is well explained and novel in that it allows curve path planning for object avoidance when manipulating granular objects. Results also show NFD-curve does outperform the model-free baseline. Overall, in model based method comparison, NFD linear does outperform baselines in Table 1.
review_title: null

Meta-review: This paper learns a dynamics model for granular materials, assuming a top-view camera observation. The model proposed in the paper is a Fully Convolutional Network, which seems to perform well, but relies heavily on the top-view assumption so that the pile of granular material can be treated as a 2.5D heightfield. Manipulation of the pile is done in a myopic way by optimizing the parameters of a curvilinear trajectory, which leads to inefficient motion, as evidenced by the accompanying videos. Overall, I think the main contribution of this paper is advocating for the FCN as a structural design choice for modeling granular media. I recommend that the paper be accepted as a poster.
Decision: Accept (Poster)