Learning Granular Media Avalanche Behavior for Indirectly Manipulating Obstacles on a Granular Slope
Authors: Haodi Hu
Feifei Qian
Daniel Seita
Venue: CoRL 2024 (2024)
Keywords: Granular media
Avalanche dynamics
Legged robots.
TL;DR: In this work, we propose Granular Robotic Avalanche INteraction (GRAIN), a novel learning-based method for leveraging granular avalanche dynamics for indirectly manipulating objects on a granular slope.

Abstract: Legged robot locomotion on sand slopes is challenging due to the complex dynamics of granular media and how the lack of solid surfaces can hinder locomotion. A promising strategy, inspired by ghost crabs and other organisms in nature, is to strategically interact with rocks, debris, and other obstacles to facilitate movement. To provide legged robots with this ability, we present a novel approach that leverages avalanche dynamics to indirectly manipulate objects on a granular slope. We use a Vision Transformer (ViT) to process image representations of granular dynamics and robot excavation actions. The ViT predicts object movement, which we use to determine which leg excavation action to execute. We collect training data from 100 real physical trials and, at test time, deploy our trained model in novel settings. Experimental results suggest that our model can accurately predict object movements and achieve a success rate ≥ 80% in a variety of manipulation tasks with up to four obstacles, and can also generalize to objects with different physics properties. To our knowledge, this is the first paper to leverage granular media avalanche dynamics to indirectly manipulate objects on granular slopes. Supplementary material is available at https://sites.google.com/view/grain-corl2024/home.

Learning Granular Media Avalanche Behavior for Indirectly Manipulating Obstacles on a Granular Slope
Haodi Hu, Feifei Qian', Daniel Seital
E'qual advising
University of Southern California, United States
haodihu@usc. edu
Legged robot locomotion on sand slopes is challenging due to the com- plex dynamics granular media and how the lack of solid surfaces can hinder locomotion. promising strategy, inspired by ghost crabs and other organisms in nature, is strategically interact with rocks, debris, and other obstacles to fa- cilitate movement. To provide legged robots with this ability, present novel approach that leverages avalanche dynamics to indirectly manipulate objects on a granular slope. We use Vision Transformer (ViT) to process image repre- sentations of granular dynamics and robot excavation actions. The ViT predicts object movement, which we use to determine which leg excavation action to ex- ecute. We collect training data from 100 real physical trials and, at test time, deploy our trained model novel settings. Experimental results suggest that our model can accurately predict object movements and achieve success rate 2 80% in variety of manipulation tasks with up to four obstacles, and also generalize to objects with different physics properties. To our knowledge, this is the first paper to leverage granular media avalanche dynamics to indirectly manipulate objects on granular slopes. Supplementary material is available at attps://sitte http://sstsss..ggleeeee http://vieeggraannoorr//
Abstract:
Keywords: Granular media, Avalanche dynamics, Legged robots, Manipulation of deformable substrate.
Introduction
Legged locomotion across granular surfaces such as sand formidable challenge due factors such insufficient support offered by the sand surface and the complex dynamics of leg-sand in- teractions [1, 2, 3]. It is particularly challenging for robots to climb up steep granular slopes, as the sand could easily flow underneath robot legs due the reduced shear resistance forces [4]. Re- cent studies on obstacle-aides -ided robot locomotion show the potential for legged robots to strategically leverage large obstacles within sand, such as rocks and boulders, to traverse granular and uneven ter- rains [5, 6, 7, 8). However, such "obstacle-aided locomotion" strategies require specific leg-obstacle contact locations [5], thus ability to move rocks and boulders desired locations became es- sential for these strategies to apply. To address this challenge, this study aims to propose a method for legged robot to effectively reposition obstacles granular slopes by primarily leveraging in- direct manipulation. Prior work has shown that external disturbances on sand incline can trigger avalanche behavior [9, 10, 11], suggesting the potential for legged robot to exploit this property to advantagcously relocate obstacles on granular surface.
In this work, we propose Granular Robotic A valanche INteraction (GRAIN), novel learning- based method for leveraging granular avalanche dynamics for indirectly manipulating objects granular slope. Due lack of accurate simulators for simulating legged robots and avalanche behavior granular slopes, we do all experiments in the real world. We use RHex [12] family robot leg as an external disturbance source which performs excavation actions within a grain tank with mechanical
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Obsacces
Roboot Hag.
Gantry structure
(d
Figure I: Top: (a) our quadrupedal robot and (b) a granular slope with the rohot and obstacles; (c) our setup to collect data from one rohot leg (highlighted in yellow) manipulating obstacles, with our designed gantry system (highlighted in eyan) for moving the leg of our proposed system to manipulate obstacles on the granular slope. The three blue arrows (d) represent the change in each obstacle's 's location after applying the excavation, and the red shaded areas are targets.
the granular slope. Bottom: (d), (c), (f) an example trial
support to form slope. We also design a gantry structure enable the robot leg move different positions for performing leg excavations. We collect training data through physical interaction and deploy our trained model in the real world with a quadrupedal robot. See Figure 1 for our setup.
Accurately predicting the granular media dynamics is essential for our proposed task, but is funda- mentally challenging [1, 11, 13, 14]. address this, we leverage learning methods in this study. We place a rigid 3D-printed obstacle on the granular slope and collect data by applying continuous robot leg excavation actions to investigate avalanche behavior. We represent the granular surface state via the current depth image and the change in depth between sequential robot leg excavations, and we also use an image to represent the leg excavation action. This enables a unified image-baser input to our granular media dynamics model. We train a ViT [15] to take our proposed granular media and robot excavation action representations as input and output the object movement on the granular slope. Experiments over 30 trials suggest that our model can accurately predict object movements on a granular slope, and can generalize objects with different physics properties.
In summary, our main contributions are follows:
1. A novel problem formulation of using legged robots and leg excavation actions to indirectly manipulate obstacles to targets over complex, granular terrains.
2. GRAIN, a novel approach that uses image-based representations of granular dynamics and legged excavation actions to predict object movements on a granular surface.
Experimental results showing that a legged robot using GRAIN can move obstacles to targets, and that GRAIN outperforms an ablation and non-learning baseline.
2 Related Work
Robot interaction with granular media: Researchers have explored robotics and granular media in locomotion and manipulation contexts. For example, researchers have enabled crawling [16, 4], hopping [17, 18], running [19, 2, 3], climbing [20], and rapid turning [21] over granular media by leveraging advances in granular force models [1, 13] and machine learning methods [22, 23). We focus on the orthogonal task of leveraging the properties of granular media to manipulate objects on it. This requires some understanding of how granular media behave, which could be from direct physics analysis [24, 25, 26] or learned models [27, 28]. These models can aid common manip- ulation tasks involving granular media, which include pouring [29, 30, 31, 32, 33, 34], scooping and bulldozing [35], trenching [36, 14] or adjusting soil with plates [37]. Among relevant prior

---

work, Wang et al. [38] and Xue et al. [39] study particle-based and density field based represen- tations, respectively, to learn dynamics models for planar tabletop manipulation of granular media, and Schenck et al. [40] use image representations and train a convolutional neural network to predict changes in the granular media from scooping or pouring actions. As [40], we use image-based representations of granular media, but our actions are based on leg excavations instead of scoop- ing, pouring, or other tabletop manipulation actions. Moreover, use legs of legged robot to indirectly manipulate objects on granular media surfaces.
Locomotion over diverse terrains: Recent research has proposed methods that enable legged robots to traverse over a variety of terrains, including outdoor settings that may have sand, vege- tation, rocks, or other granular media. A promising technique is reinforcement learning coupled with advanced simulators, which can help avoid significant manual engineering [41, 42]. To better handle diverse terrains,
line of work proposes adaptation methods, either via rapid motor adap- tion [43, 44] or by encoding family of gait methods which facilities tuning to new terrains [45]. Other works study navigation over challenging terrains [46], which may involve climbing, jump- ing [47], and crawling under parkour-style settings [48, 49). While impressive, these works study locomotion over terrains that are much sturdier than our granular surface. Furthermore, they primar- ily consider locomotion, whereas we focus on manipulating objects on a granular surface.
Manipulation with legged robots: While legged robots primarily use legs to move to target location, they can also use legs for manipulation. For example, researchers have proposed methods for using legs to kick soccer balls [50, 51] and to push obstacles [52, 53]. A legged robot can also use two legs to stand up to better enable other legs to press against higher objects such as door buttons [54]. Other works mount an arm on top of a legged robot, and leverage methods such as optimization [55, 56] or machine learning [57, 58, 59] allow the arm to manipulate objects. These works legged robots for manipulation via direct contact with object. To knowledge, our work is the first to show legged robot indirectly manipulating an object make it reach desired pose. To do this, the robot adjusts granular surface that supports the object.
3 Preliminaries and Problem Statement
We consider the RHex [12] family of legged robots, with DOF for each leg. We generate an excavation action, per leg, by commanding the leg rotate constant angular speed for one circular cycle. We assume the robot lies granular (sand-like) surface which has slope of Φ degrees. This surface has K > 1 rigid obstacles, and we indicate their respective positions at given time & as {s ..... s ( The task is move all obstacles from their initial locations to pre-specified desired locations (p ,..,,,.. ..).,, After doing this, we may also want to move the obstacles to second target location, and we indicate these optional (per-obstacle) target locations as (q ......gK) J. Here, each 5; E R²2 (()) E 1K and (k)) E 2 for & E 21, J. since we specify 2D positions over an image of the granular surface. For notational convenience, when K overhead camera, which provides (H x W) depth image observations x,. The objective is to learn policy which produces leg excavation action a, at time f, where the robot rotates one of its legs. A trial consists of executing the robot's policy until termination criteria. We evaluate a trial's performance by averaging the mean absolute error (MAE) distance among all obstacle positions and their respective target positions. We also use MAE to evaluate models which predict where obstacles move based on robot actions.
may suppress the superscript (A:). We assume access
4 Approach: GRAIN
4.1 Image Representations of Granular Dynamies
Prior work has shown that external disturbances can trigger avalanche behavior on granular slopes [60]. Inspired by this, we aim to leverage robot leg excavation actions to cause avalanche behaviors to move obstacles desired locations. Intuitively, the robot leg excavation location af- fects the avalanche area, and an obstacle's relative position to the excavation affects how much
3

---

Action ag
Observation
X, and Ax;
VIT
Ri
Ohservation Ax
VIT
VIT
Actions
a/
State
Action ((ax)
(a) Model Training
(b) Policy Execution
Figure 2: Overview of GRAIN. (a) The VIT has three inputs: the depth image of the granular surface X the change in depth before and after excavation Ax., and image representation output of the VIT the prediction of the obstacle's post-excavation location which is a 2D vector. (b) We use the trained VIT to predict object movements based on leg excavation actions. The VIT combines the current observation with candidate actions as input and prediets corresponding object movement. The VIT considers one obstacle output; see Sec. 4.3 for handling obstacles at test time. See Sec. 4 for details on notation.
action [(a)). The
it is influenced by the avalanche. Due to the complexities of modeling avalanche dynamics and obstacle movements on the granular surface, we learn an action-conditioned dynamics machine learning model to predict obstacle movement. Similar learned models have predicted complex physical interactions for planning robot manipulation [61, 62, 63], suggesting their utility for our task. To represent the obstacle and the granular slope surface, we use top-view depth image, X Furthermore, during preliminary experiments, we ob- served that successfully relocating obstacles often re- quired consecutive excavations at the same location. See Figure for a visualization of grain flows with sequen- tial excavations. To highlight this physical finding our model, we use the change in the depth images of the gran- ular slope surface between sequential excavation actions, denoted as Ax; X, X for O, with Ax, 0 (all pixels zero) if O. We also introduce an image rep- resentation of robot leg excavation locations, which en- ables the action input to be spatially aligned with X, and Ax,- This may help the network better learn the avalanche behavior compared to the action is processed separately and concatenated with downstream visual features. We discretize excavation locations to 15 locations in 3 grid (see Figure 5). We plot white square with side length the same the robot leg length a black background to represent the robot leg excavation location, I [a.)), where a, is the excavation location the 2D planar surface, and / is function that generates the RGB image representation of
Leg
Leg
Figure 3: Granular flow for sequential ex- cavations. Red vectors represent the change of particle positions between two frames.
4.2 Training Objective for the Dynamics Model
We train a VIT to learn granular avalanche dynamics from C-shape robot leg excavations. The VIT predicts one obstacle movement on the granular slope, and we deal with multiple obstacles at test-time by repeatedly querying the model (see See. 4.3). The training loss L is defined as:
C(x:: Axi, a 81+1) [o Axi, I(a:)) s ll
(1)
where Fg is the VIT parameterized by that takes, as input, a channel-wiss concatenation of three spatially-aligned images: the depth image x,, the change in depth Ax, and the action representation ((a.). The ViT outputs the predicted post-excavation obstacle position $ Fo(x,, Ax,, T(a,)), which we compare with the ground truth obstacle position S on the inclined 2D surface time t + 1. Since the VIT predicts continuous values, modify the VIT's default MLP classification header with an MIP regression header. See Figure 2 (left) for an overview of training.

---

4.3 Leg Manipulation Policy
We propose a greedy strategy for manipulation, which means the robot leg performs the excava- tion action at the location that has maximum obstacle movement projection on the current line connecting the desired location and obstacle center toward the target area, described in Eq. 2:
a arg max e (Fo Axt; [(a))) st), a.EA
where A is the set of 15 possible actions (see Figure 5), and cf is 2D unit vector that points from the obstacle center to the target location at time (see Figure 4, target is the shaded red square). Parameters such as leg insertion depth and the angle of the excavation could also offer additional terrain terrain-mantnn manipulation opportunities. As an initial step, focus demonstrating the effect of leg excavation location. Prior work has shown that greedy policy for planning can be useful for object relocation [64], and we hypothesize the same may be true in our setting.
et
Our model is trained using robot excavation actions with a single obsta- cle. To use the model with multiple obstacles, we treat multiple obstacle movement prediction
Figure Example of e, vector, pointing the ob- stacle's target.
collection of independent predictions of sin- randomly select an obstacle
gle obstacle movement. In particular, and mask out other obstacles on the depth image input, where we use a window that has length 3 times the obstacle radius to compute the average pixel values in the windows and replace the pixel value of the obstacle with the computed averaged pixel value. Furthermore, we mask out other ob- stacles in Ax,, where we set the pixel values to 0 corresponding to other obstacle positions. We repeat this process for all obstacles and get the predictions of obstacle movements. We mod- considers the of obstacles projected
ify our manipulation policy fit the task; the policy movement lines that connect their centers to their desired locations described in Eq. 3:
Σ e (Fo(x;, Axi, I(a+)) 8z),
a = arg max a.EA
where images X, and Ax, are masked versions of x, and Ax,-
5 Experiment Setup
Existing simulators used in bamming-based legged robot manipulation research, such PyBul- let [65], MuJoCo [66], or IsaacGym [67], do not support realistic robot interaction on granular media surfaces. Thus, we do all data collection, training, and experiments directly in the real world.
Experiment environment: Figure illustrates our experiment setup. The main structure of the testbed is granular trackway (60 L cm W c0 D) filled with 6 6m plastic BBs (Matrix Tactical Systems). The 6 mm particles
chosen they behave rheologically similar with natural sand and soil [68, 1, 69], while the simpler geometry and larger size facilitate image- based model training. The granular trackway can be tilted up to 35 degrees to emulate a wide variety of sand slopes [11] in natural environments. To study the avalanche dynamics and object movement upon different leg excavation actions, we build gantry system with two linear actuators (one moves along the axis and another along the y axis) move C-shape robot leg on a 2D surface above the granular slope. The C-shape robot leg has diameter of 6.0 cm and width of 2.0 cm, and the rotation center of the leg 1.0 em above the initial granular slope surface. The rotation frequency of the robot leg is fixed at 0.33 Hz. This is sufficiently above the granular surface, so the robot avoids touching it while transitioning between consecutive excavation actions. We mount an RGBD camera (Intel RealSense 435-i) above the granular slope to record the granular flow and obstacle movement.
Data collection: We collect dataset of 100 trials, where each trial has 10 excavation actions. The time interval between two consecutive excavation actions is 12s to enable the robot leg to transit to different locations. Before each trial, human operator manually smoothed the granular media to
5

---

Figure 5: Obstacle movement with different excavation actions, visualized with colored squares (5 actions per image above). These 5 x 3 15 colored squares are the 15 discretized excavation actions consider. The corresponding solid color circles and empty color circles the trained model's predicted obstacle positions experiment-menne measurred obstacle positions, respectively. (This figure is best viewed zoomed-in.)
(roughly) even granular slope with an inclination angle O = 18 degrees. This inclination angle is close to the angle of repose [70] of the granular material used in our work, which facilitates the study of the avalanche dynamics. Once the granular surface is prepared, the human placed 3D-printed (PLA) obstacle on the granular surface different locations relative to the leg. For all 100 trials, semi-spherical obstacle with 5 em diameter used. The RGBD camera has video streaming rate of 15 Hz, and it collects 640x480 RGBD images after every excavation action. The ground truth obstacle movement is calculated based on the post-excavation RGBD image. Among the 100 trials, 36 use the same excavation location but different initial obstacle positions, 30 use the same initial obstacle position but different excavation locations, and 34 vary both the initial obstacle positions and excavation locations for each action. We use this data to train Fg-
Evaluation: For each trial, after the human places the obstacles, computer program randomly selects target location(s) for each obstacle. Targets can be anywhere in the robot excavation action space, long they do not overlap with each other. Then, the program randomly selects method among a set of methods we test (GRAIN, baseline, or an ablation, see Sec. 6.2) for manipulation, to reduce human bias in making initial settings easier for our method. We evaluate our dynamics model and manipulation outcomes based on the prediction error for each excavation action, using Mean Absolute Error (MAE). In test trials, we compute performance based on the final distance between the obstacle center and its desired location. The success threshold is below 2.5 cm (the radius of the obstacle) which measure via converting pixel distances in images centimeters. We also use the average error between prediction and ground truth as a performance evaluation.
6
Experiment Results and Discussions
6.1 Model Performance on Predicting Obstacle Movements
To evaluate our model's performance predicting obstacle movements, we place an obstacle on an undisturbed granular slope and test 15 excavation action locations; we reset the obstacle to the same spot after each action. See Figure 5 for comparison between the predictions and the ground truth locations. The MAE for these 15 excavation actions is 1.13 cm, below our 2.5 em threshold. Based on the promising MAE results, we use our trained model for planning in Sec. 6.2 and Sec. 6.3.
6.2 Single Leg Manipulation Performance
We evaluate GRAIN in real-world experiments with a single leg using four types of tasks:
1. Single obstacle with a single task: Using K = 1 obstacle, with a target position p.
Single obstacle with sequential tasks: Using K
with an additional (second) target q.
Multiple obstacles: Using K 4 obstacles with targets (p(2), (2) .p (3) ()))))()))
4. Unseen obstacle: Using K obstacle with target position p. but where the new obstacle has star shape and weighs twice as much as the standard obstacle we use.

---

Subtask1
Subtask2
Figure 6: Single leg manipulation results, showing one trial of "Single obstacle with sequential tasks". The red shaded boxes the obstacle targets. The green dots the predicted post-excavation locations of obstacles after the leg performs excavation its location.
We compare GRAIN against a baseline and an ablation. Our baseline is an algorithmic, non-learning manipulation strategy. The baseline randomly selects an obstacle, and the robot performs the exca- vation at the closest available location to the target until the obstacle has met one of two termination criteria (see next paragraph). Then, this baseline randomly selects one of the remaining obstacles to manipulate and repeats until all obstacles image representation of the robot excavation action to the successful training of our model. We train a ViT with awwer-dimensional, 2D vector representation of the robot excavation action. This cannot be directly channel-wise combined with the depth-based image observations, so we instead input it to the regression header of our VIT.
selected. Our ablation investigates the importance of
manipulation trial terminates upon either of these conditions: (i) none of the excavation actions take obstacles closer to their targets, or (ii) all obstacles have small accumulated movement (S 0.5 cm) over 3 sequential excavations. In (i), for the baseline method, we terminate when observe that the previous excavation action took the obstacle away from its target or reached it.
Results from Tab. 1 suggest that GRAIN outperforms the baseline and the vector representation ablation. GRAIN has success rate 2 80% on all manipulation tasks, while the baseline has trouble with the "Multiple Obstacles" task, and the ablation has trouble with the "Unseen obstacle" task. The baseline particularly struggles when manipulating multiple obstacles (20% success), it does not consider other obstacle movements when manipulating one obstacle. The ablation's granular dynamics model has higher (i.e., worse) MAE in the predictions. which results in lower success rates versus GRAIN. We show one trial each of "Single obstacle with sequential tasks" and "Multiple obstacles" Figure 6. refer the reader to the supplementary website for videos.
Success rate (%)
Task
Method
MAE (cm)
MAE (cm)
Baseline
N/A
1.78( 0.96)
80%
Single obstacle with single task
Vector representation GRAIN
2.34( 0.94)
1.76(# 0.63)
80%
1.66( 0.62)
1.39(L 0.41)
100%
Baseline
N/A 2.51( 0.96)
2.24(# 1.12)
60%
Single obstacle with sequential tasks
Vector representation
2.17(# 0.72)
60%
GRAIN
1.78(t 0.62)
1.42(t 0.45)
100%
Baseline
N/A 2,89(1 L.11)
5.12(# 3.44)
20%
Multiple obstacles KK
Vector representation
2.21(# 0.88)
60%
GRAIN
2.21( 0.91)
1.85( 0.74)
80%
Baseline
N/A 2.67( 1.42) 1.96( 0.90)
4.34( 2.87)
60%
Unseen obstacle
Vector representation
2.49( 1.38)
40%
GRAIN
1.91 0.73)
80%
MAE for model predictions versus ground truth. MAA for final obstacle positions versus target positions.
Table I: We compare the quantitative performance of method (GRAIN)
alternative methods four
manipulation tasks. report MAP in two columns, both using format: Mean(+: Standard Deviation), over 5 trials each. The "Baseline" method involves no prediction, hence the "N/A" in the first MAE column.
7

---

6.3 Quadruped Robot Manipulation Experiments
We evaluate GRAIN on the quadrupedal robot with front legs as manipulators; we do not use the back legs since the robot body would likely block obstacles. Specifically, we use the same trained model as in Sec. 6.2 but we query the model the 2 excavations where the legs are located, instead of the full set of 15. We show one trial in Figure 7. Results in Tab. suggest that GRAIN achieves higher performance (i.e., lower MAE) than the baseline or vector representation methods.
Success
Task
Method
MAE (cm)
MAE (cm)
rate (%)
Baseline
N/A
4,56(# 2.46)
30%
Multiple obstacles (K 3)
Vector representation
3. 1.1 1.34)
2.67(1 1.12)
40%
GRAIN
2.5 1.08)
1.91 0.97)
70%
MAE for model predictions versus ground truth. 'MAE for final obstacle positions versus target positions.
Table 2: We compare the performance of GRAIN and alternative methods on the multiple obstacles task (with 3 obstacles) using the quadrupedal robot, in similar format as Tab. 1, except statistics are over 10 trials each.
Figure 7: One trial of a quadruped robot manipulating three obstacles to reach targets (indicated with red shaded boxes). The yellow boxes highlight the leg that performs the excavation action at each step, and the green dots are the predicted post-excavation locations of obstacles after the leg performs the action.
6.4 Failure Cases and Limitations
common failure of GRAIN with multiple obstacles occurs when the robot leg moves one obstacle to its target while simultaneously moving other ob- stacles too far from their targets. See Fig. for an example, where obsta- cles are already at their target locations (red shaded boxes), but the robot has trouble moving the third obstacle to its target (white arrow and green shaded box). The rightmost obstacle has reached its target, and further excavations will move it away from its target. Our manipulation policy only predicts ob- stacle movements for one step, and thus can have difficulty with tasks that require multiple-step planning, which we plan to address future work.
Figure 8: The arrow shows the robot cannot move the obstacle to its target (green square).
Conclusion
In this work, we present GRAIN, a method for legged robot to indirectly manipulate obstacles on granular surface. Our method trains neural network to predict where obstacle a granular surface moves as the result of a legged robot's excavation action, enabling quadrupedal robot to leverage granular avalanche dynamics to relocate obstacles. To our knowledge, this is the first work to explore indirect manipulation on deformable surfaces via legged robots. As the first step towards the complex problem of indirectly manipulating obstacles via strategic granular avalanches, we used granular particles of relatively simple geometry, and primarily focused on individual obstacle move- ment prediction with minimal inter-obstacle interactions. Prior work has shown that these simpler geometry particles behavior qualitatively similar in terms of rheology and interaction force laws [1], suggesting strong potential for GRAIN to be extended to more complex scenarios in future work.

---

Acknowledgments
This work is supported by funding from the National Science Foundation (NSF) CAREER award #2240075, the NASA Planetary Science and Technology Through Analog Research (PSTAR) pro- gram, Award # 80NSSC22K1313, and the NASA Lunar Surface Technology Research (LuSTR) program, Award # 80NSSC24K0127. The authors would like to thank Luke Cortez for helping with preliminary data collection, and Vedant Raval for helpful writing feedback.
References
[1] C. Zhang, and D. I. Goldman. A terradynamics of legged locomotion on granular media. science, 33966126):1408-15 1412, 2013.
[2] F. Qian, T. Zhang, C. Li, P. Masarati, A. M. Hoover, P. Birkmeyer, A. Pullin, R. S. Fearing, and D. Goldman. Walking and running on yielding and fluidizing ground. Robotics: Science and Systems, page 345, 2013.
[3] Qian, T. Zhang, W. Korff, P. B. Umbanhowar, R. J. Full, and D. Goldman. Principles of appendage design in robots and animals determining terradynamic performance flowable ground. Bioinspiration & biomimetics, 10(5):050014, 2015.
[4] H. Marvi, C. Gong, N. Gravish, H. Astley, M. Travers, R. L. Hatton, J. R. Mendelson III, H. Choset, D. L.. Hu, and D. I. Goldman. Sidewinding with minimal slip: Snake and robot ascent of sandy slopes. Science, 34666206):224-22,, 2014.
[5] Qian and Goldman. Anticipatory control using substrate manipulation enables trajectory control of legged locomotion heterogeneous granular media. In Micro-and Nanotechmology Sensors, Systems, and Applications VII, volume 9467, page 94671U. International Society for Optics and Photonics, 2015.
[6] Qian and D. E. Koditschek. An obstacle disturbance selection framework: emergent robot steady states under repeated collisions. The International Journal of Robotics Research, 2020.
[7] K. Chakraborty, H. Hu, M. D. Kvalheim, and Qian. Planning of obstacle-aided navigation for multi-legged robots using sampling- baseed method over directed graphs IEEE Robotics and Automation Letters, 7(4):8861-886) 2022.
[8] H. Hu and F. Qian. Obstacle-aided trajectory control a quadrupedal robot through sequential gait composition. IEEE Transactions on Robotics, pages 1-15, 2024.
[9] G. Barker and A. Mehta. Two types of avalanche behaviour in model granular media. Physica
A: Statistical Mechanics and its Applications, 283(3-4):328 22662, 2000.
[10] S. P. Pudasaini and K. Hutter. Avalanche dynamics: dynamics of rapid flows of dense granular avalanches. Springer Science & Business Media, 2007.
[11] N. Gravish and D. I. Goldman. Effect of volume fraction on granular avalanche dynamics. Physical Review E, 90(3):032202, 2014.
[12] U. Saranli, M. Buehler, and D. E. Koditschek. Rhex: A simple and highly mobile hexapod robot. The International Journal of Robotics Research, 20(7):616 631, 2001.
[13] S. Agarwal, D. I. Goldman, and K. Kamrin. Mechanistic framework for reduced-order models soft materials: Application to three-dimensionall granular intrusion. Proceedings of the National Academy of Sciences, 1204)102214017120, 2023.
[14] Q Yu, C. Pavlov, W. Kim, and A. M. Johnson. Modeling wheeled locomotion in granular media using 3d-rft and sand deformation. Journal of Terramechanics, 115:100987, 2024.

---

[15] A. Dosovitskiy. L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[16] N. Mazouchova, P. B. Umbanhowar, and D. I. Goldman. Flipper-driven terrestrial locomotion of sea turtle-inspired robot. Bioinspiration & biomimetics, 8(2):026007, 2013.
[17] J. Aguilar and D. I. Goldman. Robophysical study of jumping dynamics on granular media. Nature Physies, 12(3):278-285 2016.
[18] A. H. Chang, C. M. Hubicki, J. Aguilar, D. I. Goldman, A. D. Ames, and P.A. Vela. Learning to jump in granular media: Unifying optimal control synthesis with gaussian process-based regression. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
[19] C. Li, B. Umbanhowar, H. Komsuoglu, D. E. Koditschek, and D. Goldman. Sensitive dependence of the motion of legged robot on granular media. Proceedings of the National Academy of Sciences, 106(9):3029-300 2009.
[20] A. Karsai, D. Kerimoglu, D. Soto, S. Ha, T. Zhang, and D. Goldman. Real-time remodeling of granular terrain for robot locomotion. Advanced Intelligent Systems, 4(12):22001 19, 2022.
[21] D. Kerimoglu, D. Soto, M. L. Hemsley, J. Brunner, S. Ha, T. Zhang, and D. I. Goldman. Learning manipulation of steep granular slopes for fast mini rover turning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 16985-16990. IEEE, 2024.
[22] Lee, J. Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47))eabc5986, 2020.
[23] S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo. Learning quadrupedal locomotion on deformable terrain. Science Robotics, 8(74)eade2256,, 2023.
[24] H. Hinrichsen and D. E. Wolf, editors. The Physics of Granular Media. Wiley-VCH Verlag GmbH & Co. KGaA, 2004.
[25] D. L. Henann and K. Kamrin. A predictive, size-dependent continuum model for dense gran- ular flows. Proceedings of the National Academy of Sciences, 110(17), 2013.
[26] K. Kamrin and G. Koval. Nonlocal constitutive relation for steady granular flow. Physical Review Letters, 108(17), 2012.
[27] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying. J. Leskovec, and P. W. Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning (ICML), 2020.
[28] N. Tuomainen, D. Blanco-Mulero, and V. Kyrki. Manipulation of granular materials by learn- ing particle interactions. In IEEE Robotics and Automation Letters (RA-L), 2022.
[29] L. Rozo, P. Jiménez, and C. Torras. Force-based robot learning of pouring skills using para- metric hidden markov models. In 9th International Workshop on Robot Motion and Control, pages 227-232. IEFE, 2013.
[30] M. Cakmak and A. L. Thomaz. Designing robot learners that ask good questions. In ACM/IEEE International Conference on Human-Robo Interaction (HRI), 2012.
[31] A. Yamaguchi and C. G. Atkeson. Differential dynamic programming with temporally de- composed dynamics. In CEEE-RAS 15th International Conference Humanoid Robots (Hu- manoids), 2015.
10

---

[32] A. Yamaguchi and C. G. Atkeson. Differential dynamic programming for graph-structured dynamical systems: Generalization of pouring behavior with different skills. In IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), 2016.
[33] A. Yamaguchi and G. Atkeson. Neural networks and differential dynamic programming for reinforcement learning problems. In IEEE International Conference on Robotics and Automa- fion (ICRA), 2016.
[34] B. Wu, F. Xu, Z. He, A. Gupta, and P. K. Allen. Squirl: Robust and efficient learning from video demonstration of nonghhoizoon robotic manipulation tasks. In IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), 2020.
[35] D. Millard, D. Pastor, J. Bowkett, P. Backes, and G. Sukhatme. Granular gym: High perfor- mance simulation for robotic tasks with granular materials. In Robotics: Science and Systems (RSS), 2023.
[36] C. Pavlov and A. M. Johnson. Soil displacement terramechanics for wheel-based trenching with planctary rover. In 2019 International Conference Robotics and Automation (ICRA), pages 4760-4766. IEEE, 2019.
[37] M. Kobayakawa, S. Miyai, T. Tsuji, and T. Tanaka. Interaction between dry granular materials and an inclined plate (comparison between large-scale dem simulation and mice-dimensional wedge model). Journal of Terramechanics, 90:3-10, 2020.
[38] Y. Wang, Y. Li, K. Driggs- Campbel, L. Fei-Fei, and Wu. Dynamic-Resolutinn Model Learning for Object Pile Manipulation. In Robotics: Science and Systems (RSS), 2023.
[39] S. Xue, S. Cheng, P. Kachana, and D. Xu. Neural Field Dynamics Model for Granular Object Piles Manipulation. In Conference on Robot Learning (CoRL), 2023.
[40] C. Schenck, J. Tompson, S. Levine, and D. Fox. Learning robotic manipulation of granular media. In Conference on Robot Learning (CoRL), 2017.
[41] W. Yu, D. Jain, A. Escontrela, A. Iscen, Xu, E. Coumans, Ha, Tan, and T. Zhang. Visual-locomotion: Learning to walk on complex terrains with vision. In Conference on Robot Learning (CoRI), 2021.
[42] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. In Robotics: Science and Systems (RSS), 2021.
[43] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems (RSS), 2021.
[44] A. Kumar, Z. Li, J. Zeng, D. Pathak, K. Sreenath, and J. Malik. Adapting rapid motor adap- tation for bipedal robots. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.
[45] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning (CoRL), 2022.
[46] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. Conference on Robot Learning (CoRL), 2022.
[47] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, Kim, and P. Agrawal. Learning to jump from pixels. In Conference on Robot Learning (CoRL), 2021.
[48] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference Robot Learning (CoRL), 2023.

---

[49] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
[50] Y. Ji, G. B. Margolis, and Agrawal. Dribblebot: Dynamic legged manipulation in the wild. In IEEE International Conference Robotics and Automation (ICRA), 2023.
[51] Y. Ji, Li, Y. Sun, X. B. Peng, S. Levine, G. Berseth, and K. Sreenath. Hierarchical rein- forcement learning for precise soccer shooting skills using a quadrupedal robot. In IEEE/RS) International Conference on Intelligent Robots and Systems (IROS), 2022.
[52] A. Rigo, Y. Chen, S. K. Gupta, and Q. Nguyen. Contact optimization for non non-prehensile loco- manipulation via bierarchical model predictive control. IEEE International Conference on Robotics and Automation (ICRA), 2023.
[53] M. Sombolestan and Q. Nguyen. Hierarchical adaptive loco- cero-manppulation control for quadruped robots. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[54] X. Cheng, A. Kumar, and D. Pathak. Legs as manipulator: Pushing quadrupedal agility beyond locomotion. In IEEE International Conference on Robotics and Automation (ICRA), 2023.
[55] H. Ferrolho, V. Ivan, W. Merkt, I. Havoutis, and S. Vijayakumar. Roloma: Robust loco- manipulation for quadruped robots with arms. In Autonomous Robots (AURO), 2023.
[56] A. Rigo, M. Hu, S. K. Gupta, and Q. Nguyen. Hierarchical optimization- based control for wodle-body loco-manipulation of heavy objects. In IEEE International Conference on Robotics and Automation (ICRA), 2024.
[57] Y. Ma, F. Farshidian, T. Miki, J. Lee, and M. Hutter. Combining dearning-based locomotion policy with based manipulation for legged mobile manipulators. In IEEE Robotics and Automation Letters (RA-I.). 2022.
(58) Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Conference on Robot Learning (CoRL), 2022.
[59] M. Liu, Z. Chen, X. Cheng, Ji, R. Yang, and X. Wang. Visual whole-body control for legged loco-manipulation. arXiv preprint 4rXiv:2402.16776, 2024.
[60] A. Daerr and S. Douady. Two types of avalanche behaviour granular media. Nature, 399 (6733):241- -233, 1999.
[61] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
[62] A. X. Lee, R. Zhang, F. Ebert, P. Abbeel, C. Finn, and S. Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.011 01523,, 2018.
[63] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Visuospatial foresight for multi-step, multi-task fabric manipulation. In Robotics:
Science and Systems (RSS), 2020.
[64] P. Agrawal, A. V. Nair, Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Ex- periential learning of intuitive physics. In Neural Information Processing Systems (NeurIPS), 2016.
[65] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybullet. org, 2016-2021.
[66] E. Todorov, T. Erez, and Y. Tassa. MuloCo: A Physies Engine for Model-Based Control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
12

---

[67] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning. arxiv preprint arXiv:2 10470, 2021.
[68] R. D. Maladen, Y. Ding, C. Li, and D. I. Goldman. Undulatory swimming in sand: subsurface locomotion of the sandfish lizard. science, 325(5938):314-316 2009.
[69] R. Finn, M. Li, and S. V. Apte. Particle based modelling and simulation of natural sand dynamics in the wave bottom boundary layer. Journal of Fluid Mechanics, 796:340-385, 2016.
[70] R. Albert, Albert, Hornbaker, P. Schiffer, and A.-L. Barabási. Maximum angle of stability in wet and dry spherical granular media. Physical Review E, 56(6):R6271, 1997.
13

---

A Additional Details of GRAIN
A.1 Algorithm for Image Representation of Robot Excavation Action
To highlight the relationship among the robot excavation action, avalanche behavior and obstacle movement, we introduce our image representation of the excavation actions in Sec. 4.1. The pseu- docode in Alg. shows how we obtain this representation. We show several examples the image representation in Figure 9, for different locations of the rohot leg.
Algorithm Image Representation of Robot Excavation Action
Require: White square size A, coordinates of robot excavation action location = (Iz, Vc)
Ensure: Output image of size (H, W)
1: O x- Zero matrix of size (H, W)
2: Sstart - max(0,
3: Matart -- max (0, Y/ A
4: Zend min H, It +
5: Beand f min W yt + 2
6: for Zstart to Zend do
7: for j == Usturt to Yend do
8: O 255
9: end for
10: end for
Figure 9: Additional examples image representations excavation actions. Top: RGB images of robot
excavation actions. Bottom: The corresponding image representations of robot excavation actions.
A.2 Algorithm for Masking Obstacles in x,
Sec. 4.3 discusses how we handle multiple obstacles. In Alg. 2, we formalize our method to mask other unselected obstacles. This lets the masked images look similar to images from the single obstacle case, which we use for training the VIT. See Figure its corresponding masked version (obtained via Alg, 2). During training, we use cv2. col colormap, package in opencv, to convert depth to RGB, shown in the figure. For visual clarity, overlay "Obstacles" and "Masked Obstacles."
for an example an image and
14

---

Algorithm Masking Unselected Obstacles in Multiple Obstacles Manipulation
Require: Input image I of size (H,W), window size B, list of obstacle center coordinates Jk=1 (
(( ys
K-1 for each coordinate list of pixels sets (obstacles) {Pk] J
Ensure: Output image of size (H, W)
1: O f-
2: for each (r,y) E {(T m1:) k=) k=1 with corresponding pixels set P do
3: Jaan + max(0, I 7
4: Mstant f max(0.
B
5: Fend min( H, B
7
6:
Yend min W, +
B
7:
S + [[ Istart
< Fend, Ustart I yend L(iseeS [[.
8:
avg 4 [ilaad "hand-Wanst (Wess Vean)
9:
for each (i,j) E P do
if I'start S < Fend and Youn S j < Vend then O + avg
11: end if
13: end for
end for
Obstacles
Masked Obstacles
1
Figure 10: Example of masked image &. The left image is an example of x, and the right image is the corresponding masked image x.. Specifically, the second to the right most obstacle is the selected obstacle and other 3 ohstacles are masked.
A.3 Neural Network and Hy Hyperparaeeter Details
Our overall data includes 10,437 images, which obtain from 100 experiment trials. We split this data into 70% for training, 15% for validation, and the last 15% for testing. The input to our ViT is an image representation stacked channel-wise into a 7 x 128 128 input. Our VIT has 46M trainable parameters, and is smaller version of the ViT-Base model [15] so that we could fit it on our GPU. We train the ViT from scratch. To apply VIT to our task, we change the last layer of the VIT so that has an MLP header that outputs 2D vector corresponding to the predicted obstacle center position on the granular slope. Moreover, used MAE as the loss function instead of cross- cross-entropy loss. We include our hyperparameters in Table 3.
have continuous prediction instead of discrete prediction,
15

---

Hyperparameter
GRAIN Value
learning rate
5e-6
weight decay
le-3
encoder layers
MLP layers
# heads
feedforward dimension
768
hidden dimension
3072
embedding dimension
input size
7x 7x128x128
patch size
16
dropout
0.1
attention dropout probability
Table 3: Hyperparameters for the Vision Transformer (ViT) we used in GRAIN.
B Additional Experiment Details
B.1 Single Leg Manipulation Results
Figure 11 shows experiment trial for three manipulation tasks: "Multiple obstacles (K 4)." "Single obstacle with single task," and "Unseen obstacle." We refer the reader Section 6.2 for description of what the tasks mean, and for example trials from other tasks. The statistics of all manipulation trials are shown in Tab. 1.
(a)
(b)
(c)
Figure 11: Single leg manipulation experiment results: (a) Multiple obstacles 4), (b) is "Single obstacle with single tasks," and (c) is "Unseen obstacle." In the above, rod shaded areas are target areas. The green dots are the prediction of post-excavation locations of obstacles after the leg performs the action.
16

---

B.2 Multiple Unseen Obstacles Manipulation
To test our trained model's generalization ability, we use obstacles with different shapes and weights in this task. Specifically, we use a star shape obstacle, a cuboid obstacle that has half the weight of the obstacles used in the training dataset, and hemisphere obstacle the same size but 4 times the weight of the obstacles used in the training dataset. All obstacles unseen obstacles the granular slope plus the obstacle we use in the training dataset total of K = 4 obstacles with random distribution and executed the manipulation policy, We show one manipulation trial in Figure 12. Our system succeeds in out of trials.
3D-printed. We place these
Figure 12: Multiple Unseen Obstacles Manipulation. Red shaded target areas. The green dots the
prediction of post-excavation locations obstacles after leg performs action its current location.
17


Reviews:
[{"review_id": "wy1g50oLWW", "reviewer": "Reviewer_fr2L", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method to manipulate obstacles on a granular slope (surface covered in granular material such as sand). The key idea is to cause avalanche behavior of the granular material using the robot’s end effector, in this case, the leg of a quadruped robot. This behavior indirectly moves the obstacle to a desired location such that the robot can find a collision-free path. The technical method is an action-condition model that predicts future obstacle movement subject to such maneuver. Both data collection and evaluation is done in a real-world environment. The data collected by repeatedly perturbing the granular slope with a leg-shaped manipulator mounted on a fixed gantry. The trained model is also transferred to a quadruped robot. The model is evaluated based on the displacement between the desired end positions of the obstacles and the positions reached.", "summary_of_recommendation": "I would be willing to change my rating if the authors could adequately address my top 3 concerns.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "See \"weakness\" section", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3M2vv2BuSu", "reviewer": "Reviewer_egEL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In their paper, the authors propose their method (Granular Robotic Avalanche Interaction - GRAIN) for using vision transformers to obtain image representations of granular dynamics and robot excavation actions. They collect data from 100 real physical trials & their results show that their model has a success rate of 80%+ to predict object movements with up to 4 obstacles. They summarise their main contributions as the following: In summary, our main contributions are as follows: (1) novel problem formulation of using legged robots and leg excavation actions to indirectly manipulate obstacles to targets over complex, granular terrains; (2) their GRAIN method - a novel approach that uses image-based representations of granular dynamics and legged excavation actions to predict object movements on a granular surface; (3) experimental results showing that a legged robot using GRAIN can move obstacles to targets, and that GRAIN outperforms an ablation and a non-learning baseline.", "summary_of_recommendation": "The paper could be interesting to the CORL community, should the authors address the questions presented in the rebuttal section. Given the efforts the authors put into carrying it out onto a real robotic platform, I believe this is achievable and would strengthen their work for publication.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* In your related work you mention: ‘In some of the most relevant prior work, Schenck et al. [36] use an image-based representation of granular media and train a convolutional neural network to predict changes in the granular media state.’ Could you elaborate more on why you decided to use a vision transformer rather than a CNN-based method? Did you try to use a CNN-based method and deemed it insufficient? It would be nice to add these results to your experimental evaluation.\n* How representative is the obstacle of real-world obstacles in this domain? From looking at the appendix, it seems that you have used very simplified methods for dealing with obstacles (opencv-based methods). Was there a reason why you did not explore other segmentation methods e.g. segment anything model that would let you generalise to more complex obstacles?\n* Did you explore your method with other terrains e.g. soil? How would it perform with a material that might be more challenging than sand? It would be great if you can add experiments/comments on this.\n* ‘Prior work has shown that a greedy policy for planning can be useful for object relocation tasks [60], and we hypothesize that the same may be true for our setting.’ - how did you conclude this? What other policy strategies did you consider and do you have any quantitative results for these?\n* ‘Since the ViT predicts continuous values, we modify the ViT’s default MLP classification header with an MLP regression header.’ - this is slightly confusing - did you just change the loss function here?\n* ‘We train a ViT to learn granular avalanche dynamics from C-shape robot leg excavations.’ - how big is your training data? Did you train from scratch or fine-tune? Could you elaborate more on how this network looked like, what was the training/evaluation performance of this model?\n\nMinor comments:\n- Refer to images as Figure rather than Fig in the main text. \n- Hyperparameters for your models are not tabulated anywhere. Please add to the appendix to make the work more reproducible.\n- Figure 2 should have the mathematical notation in the arrows and the text in the caption.\n- Figure 4 and Figure 8 captions need to be expanded.\n- The text wrapping around the figures is a bit confusing. I would advise the authors to rethink the layout of the paper. \n- The conclusion is a bit short - maybe remove the failure case image so you have more space for the conclusion?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXr2Rse3qk", "reviewer": "Reviewer_M9RF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Inspired by living organisms this work uses avalanche dynamics for indirect object manipulation on granular slopes. This paper employs a Vision Transformer (ViT) to process images of granular dynamics and robot actions. The ViT predicts object movement, guiding the robot's leg excavation actions. Trained with data from various trials, the model performs well in new and unseen settings, achieving over 80% success in various tasks with up to four obstacles and generalizing to objects with different physical properties.", "summary_of_recommendation": "I recommend accepting the paper as it presents a novel approach to robotic manipulation of granular surfaces for displacing objects, which could potentially be used for obstacle-aided locomotion purposes.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1) In Abstract: Relocate the statement  starting with \"Supplementary material ...\"\n\n2) I recommend adding the following papers to the Related Work section, as they are pertinent to learning granular terrain manipulation for locomotion purposes.\n\n    Real‐time remodeling of granular terrain for robot locomotion\n\n    Learning manipulation of steep granular slopes for fast Mini Rover turning\n\n3) I suggest providing more explanation in the text as to why the authors chose 6 mm plastic particles.\n\n4) I wonder if the movement of one leg affects multiple objects at the same time during manipulation (Fig 7)\n\n5) I suggest enhancing (and enlarging the related region) Fig 5 for clarity.\n\n6) The conclusion section does not seem to reflect the importance of the findings of the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summarizing the strengths and weaknesses pointed out by the reviewers:

Strengths:
- Well defined problem with complex dynamics that can benefit from a learning-based solution
- Visual transformer + RL is a novel solution to the problem
- Good data collection setup
- Results show the proposed method is effective
- Well-written

Weaknesses:
- Claims about prior work are incorrect (see reviewer's comments for details)
- The action space seems to discretized to be effective, but no exploration of this is done
- The object independence assumption is not well justified
- The paper does not state how the robot moves between action locations in a single trial
- Baseline/ablation is too simple to be an adequate comparison
- The granular media used (spherical plastic particles) has limited applicability to real world scenarios

Overall, the reviewers are split (1 weak reject vs. 1 weak accept + 1 strong accept). The concerns of the weak reject reviewer appear to be mostly addressable by adding content to the paper and running a small experiment. The authors should address as many concerns of the reviewers as they can.

**Update after rebuttal**: The authors' provided a rebuttal where they directly responded to the reviewers' comments and made minor modifications to their paper. None of the reviewers replied or updated their recommendations, so let us take a look and see if the rebuttal sufficiently addressed their concerns. 2 of the reviewers already recommended acceptance, so we will focus on reviewer fr2L, who recommended weak reject. The reviewer very helpfully numbered their concerns, so we will analyze each in order:

1. The authors' response was quite extensive. It appears as though they attempted to implement both the papers referenced by the reviewer. However, they were only able to partially implement 1 of them. They included a results figure in their rebuttal, but it is not in the paper. It would have been nice to include that in the paper/appendix. The metric used in this figure is training loss, not a great metric but okay (assuming it's loss on a hold-out test set). I would say, given these results aren't in the paper, this concern is only partially addressed.

2. Again, the authors added a figure in the rebuttal but not in the paper/appendix. They show a comparison of a finer discretization than they used in the paper, and the results are very similar, justifying their choice of 5cm bins. However, they only evaluate on a single row of actions, rather than the whole space. It would have been nice to see this on the entire experiment space.

3. No new results for this one, just the authors' acknowledging the limitations of assuming independence between the objects and saying that's for future work.

4. It be confusing from the paper to understand how these actions are done, but upon a close reading it is clear that the data is collected with a gantry robot, thus locomotion is not an issue.

5. The video link was updated, and the videos do help understanding the experimental setup better.

Overall, I'd say reviewer fr2L's concerns were only partially addressed. Nonetheless, given that the other 2 reviewers recommend acceptance, and reviewer M9RF strong accept, I would say this paper should be accepted.
Decision: Accept