Gentle Manipulation of Tree Branches: A Contact-Aware Policy Learning Approach
Authors: Jay Jacob
Shizhe Cai
Paulo Vinicius Koerich Borges
Tirthankar Bandyopadhyay
Fabio Ramos
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Sim-to-Real
Deformable Manipulation
TL;DR: A reinforcement learning approach using a contact detection classifier and sim-to-real transfer for gentle interaction with tree branches.

Abstract: Learning to interact with deformable tree branches with minimal damage is challenging due to their intricate geometry and inscrutable dynamics. Furthermore, traditional vision-based modelling systems suffer from implicit occlusions in dense foliage, severely changing lighting conditions, and limited field of view, in addition to having a significant computational burden preventing real-time deployment.In this work, we simulate a procedural forest with realistic, self-similar branching structures derived from a parametric L-system model, actuated with crude spring abstractions, mirroring real-world variations with domain randomisation over the morphological and dynamic attributes. We then train a novel Proprioceptive Contact-Aware Policy (PCAP) for a reach task using reinforcement learning, aided by a whole-arm contact detection classifier and reward engineering, without external vision, tactile, or torque sensing. The agent deploys novel strategies to evade and mitigate contact impact, favouring a reactive exploration of the task space. Finally, we demonstrate that the learned behavioural patterns can be transferred zero-shot from simulation to real, allowing the arm to navigate around real branches with unseen topology and variable occlusions while minimising the contact forces and expected ruptures.

Gentle Manipulation of Tree Branches: A Contact-Aware Policy Learning Approach
Jayadeep Jacob Shizhe Cai Paulo Borges", Tirthankar Bandyopadhyay Fabio Ramos
University of Sydney, 1ata661, CSIRO, NVIIIA
Abstract: Learning to interact with deformable tree branches with minimal dam- age challenging due to their intricate geometry and inscrutable dynamics. Fur- thermore, traditional vision-based modeling systems suffer from implicit occlu- sions in dense foliage, severely changing lighting conditions, and limited field of view, in addition to having significant computational burden preventing real- time deployment. In this work, simulate procedural forest with realistic, self-similar branching structures derived from a parametric L-system model, actu- ated with crude spring abstractions, mirroring real-world variations with domain randomisation over the morphological and dynamic attributes. We then train a novel Proprioceptive Contact-Awar Policy (PCAP) for reach task using rein- forcement learning, aided by a whole-arm contact detection classifier and reward engineering, without external vision, tactile, torque sensing. The agent deploys novel strategies to evade and mitigate contact impact, favouring reactive ploration of the task space. Finally, we demonstrate that the learned behavioural patterns can be transferred zero-shot from simulation to real, allowing the arm to navigate around real branches with unseen topology and variable occlusions while minimising the contact forces and expected ruptures.
Website: https://sitte google. com//viewpcap///eeeeeeeeeeeeeeee
Reinforcement Learning, Sim-to-Real, Deformable Manipulation. Robot-vegetation interaction
Keywords:
Introduction
Active interaction with natural deformables, such as tree branches, plants, and grass, has been persistent challenge in robotics; however, the potential benefits are immense for diverse fields, in- cluding agriculture, autonomous navigation, and de-mining. The agriculture industry, in particular, benefits from automation as limited labour resources and excessive wastage threaten sustainable farming practices.
Robotic interaction with the natural environment is complex due to various factors. First, image- based perception, a cornerstone of deformable manipulation in labs, has limited success outdoors [1]. as moving branches, occluding foliage, and poor illumination disrupt camera vision. Second, the complex geometry and non-linear dynamics of natural entities limit model-based methods. Ad- ditionally, the deformable behaviour tree branches necessitates reactive control policies that gen- eralise to unseen scenarios. On the other hand, training model-free learners for long trajectories is computationally expensive, particularly with vision-derived states, while collecting real-world sam- ples is complex. Vision techniques that work well with rigid objects inadequate for deformable objects [2] like vines and stems. Vision systems also fail to assess rigidity where stress-induced deformation is non-uniform. Consequently, state-of-the-art crop harvesting systems, predominantly vision-based planners, have success rates from 50% to 75% with partial occlusions, dropping to 5% when targets are fully occluded [3].
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

While collision avoidance is conventional wisdom, contact is inevitable in messy environments; thus, minimising impact cost, both in plant damage and arm torques, is more prudent. Deep rein- forcement learning (RL) on powerful GPU-based simulators is a safer, sample-efficiet, and cost- effective option for contact-rich tasks [4][5]. However, transferring policies from simulation to real- ity, known as the sim-to-real problem [6], is notoriously hard due discrepancies between the two. Domain Randomisation, referring introducing perturbations in simulation parameters, helps ere- ate robust models generalising well to reality. Effective sim-to-real transfer and path planning with deformable contacts require realistic simulation of intricate geometry and deformation parameters to represent real-world occlusion patterns and interaction behaviour.
Given these challenges, this work proposes the following approach: First, we draw inspiration from plant morphology using the parametrie L-system (Lindenmayer system) [7] simulate realistic branching structures coupled with dynamics from crude spring actuation [8]. We domain randomise over both the L-system and dynamic spring parameters to generate a procedural forest for training our Proprioceptive Contact-Aware Policy (PCAP) for a reaching task, targeting, for instance, pruning locations, ripe fruits, or diseased regions. The PCAP states include only observations from proprioceptive sensors, not including vision or external torque sensors in favour of a blind reach policy. An independent classifier predicts contact from internal sensor measurements, which aids reward engineering. Finally, we perform zero-shot sim-to-real transfer by extending techniques from [4] operate real branches with varying dynamics. With just thirty minutes of simulation training on an RTX 4090, novel strategies emerge, allowing the arm to explore the task space while minimising contacts and contact impact. These strategies and the reactive behaviour translate well to the real world, where branch topology, obstruction patterns, dynamics parameters, and contact forces are all unknown. To summarise, our contributions are as follows:
1) We devise procedural forest generator based the L-system formalism and Domain Randomi- sation, where the individual trees are faithful the real world in geometric complexity, dimensional accuracy, and occlusion patterns.
2) We design an RL-based framework trained in simulation, which derives observations and rewards from an independent collision detection classifier and proprioceptive sensor measurements only. 3) We demonstrate that the trained reactive policy and the learned contact minimisation strategies can adapt zero-shot from simulation to real to navigate unseen branch topology and contact patterns with the help of few engincering optimisations.
2 Related Works
This section reviews advancements in sim-to-real transfer, deformable manipulation and robot-plant interaction. Sim-to-real transfer essential for applying iouulatioo-trainee policies to real-world tasks. Many studies focus on deformable object manipulation and contact-rich tasks. For instance, the IndustReal framework [4] used domain randomisation to bridge the sim-to-real gap. This ap- proach varied simulation parameters, improving policy adaptability and performance in transferring complex assembly tasks. [2] used Bayesian methods and parameter inference to manipulate de- formable objects while [9] used sophisticated steering needles to operate inside a deformable en- vironment, human lungs. [6] provided a survey on sim-to-real transfer, categorising methods into domain randomisation, domain adaptation, and imitation learning, highlighting their combined ef- fectiveness. Dextreme [5] and Shadow Hand [10] frameworks focus on agile in in-hand manipulation, using high-fidelity simulations and comprehensive RL regimes for successful sim-to-real transfer.
Photo-realistic tree representations have been proposed with L-System [7][1]][12), using polygons [13][14], or with space colonisation algorithm [15]. Furthermore, plant simulations for physical in- teractions are implemented with expensive FEM models [16] or using cheaper mass-spring systems [17][8]. From a learning perspective, [18] explored an RL-based system aided by vision for known plant topologies, emphasizing the need for detailed prior knowledge, often impractical in dynamic environments. [19] developed graph representations to predict interactions between robots and plant structures but restricted to simulation where the node positions are known apriori.

---

Policy learning for contact-rich tasks typically relies on external force/torque (F/T) sensors [20], tac- tile sensing [10], or vision [4]. These add-ons work well for contacts localised to the end-effector, but with plant manipulation, contact can occur throughout the arm. Contact-aware reaching in plants using whole-arm tactile sensors presented in [21], while internal sensors are used [22], but in model-based indoor setting. In contrast, we use proprioceptive measurements and model context. Accurate models of tree branch dynamics have been proposed [8], using internal torque sen- sors to estimate parameters like stiffness and damping. More broadly, [3] surveyed intelligent robots for fruit harvesting, identifying trends and challenges, including the need for advanced perception systems and multi-modal sensors to improve accuracy and reliability in agricultural robotics.
3 Approach
3.1 Shape Representation & Dynamies
L-system: The L-system formalism [7, 23] uses formal grammar theory and fractal geometry to model the morphology of plants and other organic structures like algae. A parallel rewriting process updates morphological attributes by applying production rules, starting from an axiom and operat- ing on a fixed alphabet of symbols. The resulting sequence of strings can be interpreted using turtle graphics [11], which moves in 2D or 3D space by attributing specific meanings to each symbol, such move-foreee forward push-pop from queue, to generate graphical structures akin to real-world ganic growth patterns. The parametric L-system associates parameters with symbols to diversify and control limb generation. Formally, a parametric OL-system is defined by G = (V, Σ, w, P). where V is the alphabet, Σ the parameter set, w the initial axiom, and P the production rules (pI.P2. Pa. Pa..P..... We extend the 'turtle' interpretation to physics simulators by generating independent cylindrical links that can be connected and actuated. We favour ternary branching structures over monopodial and sympodial variations, as real-world manipulation problems are more challenging with the for- mer. We experiment with four ternary structures: classes Ta, Tb, Te, & Td, from [7]. The ternary L-system used and the parameters for Tb class MB is provided in formulation (1), while the corresponding simulations are shown in Fig. (a)- For detailed rules, parameters, and growth attributes, see [7]; extensive simulations are available in our website.
De {d1 + 137.5, d2 + 137.5, + 18.95, pl A F(5)) (&(a) ((0)) (dd) 4, + 1.109, 2, + 0.14,
((5)) ((d) ((50))
(1)
((10)))0)
p2: F(I) : * + F(L L,);
w: 45A
p3: !ww) : * - vx)
Branch Dynamics: We extend the poarse-grained abstractions from [8] to model the deformable plant behaviour by approximating branch segments with L-system derived rigid cylindrical links, connected via revolute joints and actuated with pepportionallderiatttv (PD) controllers. However, unlike [8]. where the distribution of stiffness and damping parameters & {Kp, Ka} of the tor- sional maas--pprnggddapae system (Fig.2b) is estimated from real deformations, we do not perform system identification, instead experiment with two models to populate them in advance, namely: a) Rudimentary model: where Φ is fixed for given branch level (e.g., R22 in Fig. represents the second level) and it decreases exponentially for each level away from the trunk, starting from an up- per bound &.. tuned manually based real tree branch, and subject to lower bound Φi to avoid simulator instabilities. b) Beam deflection model [19]: where Kp Exy" and Ka Kp/10 /10, where E is the Young's modulus of the plant material, and (r; are the radius and length of the links.
Domain Randomisation:
We learn a robust policy despite the lack of strong vision priors by ran- domising over both the shape representation and the dynamics parameters. In the former case, we inject Gaussian perturbations (a = 0.1) to the L-system parameters Σ, which represents structural traits such as divergence angle (d1, d2). elongation rate (t,), width increase rate (V²), etc., to create a procedural forest (Fig.2b) for each ternary class, where individual instances have unique, diverse, and realistic forms. This paradigm fits neatly with distributed physics simulator choice Isaac Gym [24], where individual tree instances can be placed into each parallel environment. As for the
3

---

(b)
(d)
Figure 1: L-system derived branching structure on Isaac Gym, ternary classes Ta, Tb, Tc, & Td
(b)
(c)
Figure 2: (a) Seii--pppiiippeeee branch system from (b) Training with domain randomised forest generator (ternary class Ta). (c) Real setup: Kinova arm reaching for a previously unseen target (in red), obstructed by real tree branch of unknown topology and dynamics.
dynamics, while the deformation parameters & of the rudimentary model are randomised [25] with Gaussian 1.0), we do not perturb the Beam deflection model as the L-system shape randomi- sation already influences the downstream dynamics. Furthermore, during the policy learning phase in the simulation, randomise the reach target and the part of the tree the robot has access to.
3.2 Contact Detection
We use proprioceptive measurements to develop a reactive policy without vision guidance, specifi- cally the noisy joint torques, following a supervised learning approach similar to [26][27]. Unlike prior works, however, where local collision detection is the end goal, we incorporate online predic- tions from a pre-trained model into an RL framework for reach policy with gentle contact.
The proposed online collision detector workflow is as follows: First, execute arbitrary trajectories with a Kinova 6-DOF Jaco 2 arm, obstructing the path at regular intervals with soft touch or real branch contact, recording obstruction times. Second, the resulting time series dataset, i.c., sequences of joint velocity 8 and joint torque Tt, is annotated with collision labels (1-contact, O-no contact) for each time step f.. Next, we train a binary classifier to detect 'bumps' the signals, indicating contact. Finally, real-time predictions from the classifier are used for policy learning (section 3.3) as part of the state observation o, and reward computation rz- This classifier is deployed only during RL inference for real-world executions. In contrast, simulation training and testing use a proxy classifier to check if the net rigid body contact force measurements F; I available from Isaac Gym for all robot links, breach force threshold, [[[ fu). The contact impact threshold fu manually tuned based on real branch compliance during training.
Feature extraction for the classifier focuses on amplitude variations. disregarding the frequency spec- trum. We create sliding window features, [Te, Tt-1 Tt.2. (t-(m=1))- over the last time steps. Apart from min and max, we compute statistical moments, including mean, variance, skewness, and kurtosis for each of the six dofs separately. We also use smoothed version of Te by averaging the last & neighbours to account for high sensor noise levels. We add joint velocity values 8, (com- manded and executed) and the raw torque measurements T, for each joint to the feature set. Overall, we capture 28 trajectories with soft contacts on each of the links of the arm in 4 directions, plus 12 trajectories without obstructions. The final dataset has 90 features, with performing best. We tune hyper- parameersss manually and use two lightweight classifiers: 3-layer Neural Network and Random Decision Forest, settling on the latter for its high specificity.
10 and

---

3.3 Policy Learning
We formulate the reaching task as a discrete-time sequential decision problem where the agent at- tempts to maximise the expected discounted rewards accumulated by interacting with an environ- ment over time steps. Trraiiiooaaly, such problems are modelled as an MDP (Markov Decision Process) framework, defined by the quintuple A, R, 7), where S is finite set of possible states, A the set of actions taken by the agent, P(s'ls, a) the state transition model, R(s, a, s') the reward received on transitioning from to s' upon action and 7 [0, 1] the discount ra- tio, subject to s, E S and E A. Given the agent has access only to a noisy representation of the state, E O, actions can be sampled from a learned stochastic policy (alo)) parame- terised by the network weights w, i.c., a ~ Tw. where the learning objective was to maximise Specifically,
E 7 30=07"R(s,,, from the rl-games library[29), designed to avoid extensive destabilising updates w.r.t the policy at previous time step, by maximising clipped surrogate objective.
use Proximal Policy Optimisation(PPO112) variation
Action & Observation Space: In our discrete-time setting, each action ag corresponds to a 6-dof velocity target, clamped the arm limits. In velocity control mode, the arm essentially non- compliant to external disturbances, generating any amount of force clear obstacles, subject joint torque limits. We justify our use of velocity control over the more compliant torque/impedance con- trol with similar reasoning as mentioned earlier, viz. deploying torque/impedance control requires accurate F/T sensor measurements, plant manipulation requires whole body contact detection, and Kinova torque readings suffer from high noise perturbations.
On the other hand, our observation o consists of the robot joint pose, joint velocity, deviation of the end-effector pose from the reach target, end-effector orientation, and most crucially, the predicted contact flag from the aforementioned classifier. Notably, our observation space does not include any real-time inputs from the tree, such as branch images or measurements from visual tags. Besides the cost implications, adding any of these inputs can cause an explosion in training time in the context of a distributed simulator, not to mention the corresponding sim-to-real hurdles.
Our extensive reward-engineering nudges the arm evade plant contact, reduce impact upon contact, and minimise ruptures, thereby decreasing overall plant damage while reaching the target. We favour dense reward function, described in formulation (2), over episodic rewards for faster learning and easier credit assignment during each exploration step.
Reward Formulation:
Tra if |dell2 < 0.0125,
ra
1
d
Th
3ra
if [delle < 0.025,
Ys =
ra
if [dell2
< 0.05,
/=1
if F Sus [1.2
(0
if [F/l12 2/u,,
Tc
0 otherwise
Tp
1.2 otherwise.
Reward rt at each training time step t consists of five distinct components along with their cor- responding scaling factors, which are: a) a distance reward ra that progressively increases as the end-effector reaches closer to the target, where the distance d, R is computed as the differ- ence of the end-effector position from the target, b) bonus distance reward rà for close target proximity required for grasps, c) a smoothness reward r, on the six joint velocities, d) collision reward Ye to encourage contact evasion or prefer low impact collisions, and finally, e) a rupture avoidance reward to penalise extreme contact forces. For the proposed PCAP algorithm, we define r; = {ga. (r2 + r3) + 9. r.} re + Fp. where 9a- 9s are scaling factors. The product term with Te ensures that the agent receives distance rewards only for the steps where the contact forces are none or negligible; for instance, contact with leaves or thin limbs. Intuitively, the additional rupture penalty rp to account for cases where the agent learns apply quick bouts of extreme forces, in an otherwise smooth path, to rupture branches a strategic sacrifice acquire the long term reward of moving closer to the target through the breach, behaviour that we experimentally observe.
5

---

3.4 Sim-to-Real
Hardware Setup: Our policies are trained on a server equipped with NVIDIA RTX 4090 GPU, 40-core CPU, and 64GB of RAM, while the Kinova ROS api runs on a second workstation with a basic 8-core CPU, and 32GB of RAM, connected to the robot. They interact with each other via custom REST service that controls the latency specifications of the arm.
The input joint velocity frequency requirement for Kinova ROS API is 60Hz-100HZ due to on- board DSP that loops close to 10ms. On the other hand, the publish rate for sensor metrics (0,, Or, Qt. e.t.c,) 10Hz on the ROS topics. Therefore, to prevent oversampling at fetch, we nudge the policies during simulation training to generate actions at 10Hz; moreover, at the time of real deployment, we slice the output action o into equal segments to feed the API at higher frequency (e.g., 60Hz for 6), i.e, AO az/n, given AO E R6 for the six joints.
Segmented Steady State Error Control: Despite manual calibration, discrepancies persist in robot parameters, such as joint damping, friction, and gravity compensation values between simulation and real, and steady-state error is inevitable [4]. Moreover, accurately replicating the complex contact dynamics between the branches and arm is challenging [25][30], especially when the robot operates in velocity control mode. As an alternative parameter estimation, [4] proposes use Policy Level Action Integrator (PLAI) by integrating policy actions time, effectively applying them to the desired state sf in contrast to the current state 8t, in other words, * of @ d, of @ T(o,), where I is the state update operation following sampled action a; from the policy æ, Despite the absence of direct feedback control, except at policy evaluation with the current observation or, the authors show that PLAI acts as a continuous correction model robust to disturbances, compared to the conventional model, sH+1 = 8 @ (ot), that transitions from the current state. However, when the control action regulates joint velocity instead, we observe that the position error accumulates over time due the integral effect, which is harder control without direct feedback. Therefore, follow middle ground between PLAI and the nominal approach. We update from the last desired state for each segment within a time step but use the current state when updating across the trajectory time steps.
((:)
St if inn = 0 otherwise for i = 0, .., 0,,,,....,P X
(3)
Intuitively, formulation (3) could be viewed as local disturbance rejection over the short term, with periodic syncing to the current state to offset accumulated deviations.
The Domain Randomisation in section (3.1) detailed how introduce vari- ability in both branch shapes and the tree dynamics. On the other hand, we do not randomise the robot dynamics. Instead, we tune the robot damping and friction parameters heuristically by com- paring real torques and velocities to simulation generrted values, whereas the inertial priors are gleaned directly from Kinova specification. Furthermore, we perturb the measured contact forces in simulation with a small Gaussian noise (a = 1.) to account for classificr generalisation error in real.
Robot Parameters:
Experiments & Results
We evaluate the effectiveness of our proposed approach through the following experimental setups: Experiment 1 (Simulation Tests): Here, compare four distinct policies by ablating the observations and rewards: First, a baseline PPO that does not include a penalty for high contact forces and can, therefore, apply maximum arm torques (subject to arm torque limits) to push through branch occlusions, disregarding any plant damage. Second, a Contact Penalty Only (CPO) policy uses our reward formulation (2), but that does not take in the classifier predictions towards or. Third, a Model Predictive Control (MPC) baseline from [31], however, without prior for the tree geometry ensure fair comparison to our blind model. The fourth is our proposed PCAP approach, which combines both classifier prediction and reward formulation (2). Specific policy details presented in Table 1, where 0r, dr, gt. & d, represent the joint angle (6D), joint velocity (6D), the gripper

---

#PQ: Saseine-
Baseline- 1
2995
120
MPC: Baseline-
Baselin- -1yb4
51.84
CPO: 'Contact Penolty Only SR
CPO: "Contact Peralty Only- Fdlt
810%
PCAR *Contact Penalty "Clssiiirr SR1
PCAP: "Contct Paralty "Clossttor F
80%
Aur Contaxt Foruss IF4 (10
-Rupture Threshold Breach
-1%
SS 191
72%
71%
699009%
Rupture Threshold
62%
47%
-2
4n
40
+0%
28%
12
Colision
Collision
PPO: SR1
100
PCAP: SR
Max Joint Terques | max (Nm)
PCAP: 1T
Tul Systium Pattarn SR
73.9
71.7
65.2
58.7
190
80
75%%
*:Real Rupt une Stop
122%
08.1
96.6
86.4
Success Rate
80
10.0
-70
63.8
46.8
55.3
55.3
11.2%
4.5Nm 4.50m
+1.4%
11Nm,
+14%
91.2
89.5
73.7
86.0
16
16
1d
Tram L-Syutem Nuttem
Collision Patterns
Branching Patterns
Figure 3: (Top) Exp 1: Comparison of Success Rates (SR) and Average Contact Forces across four policies: PPO, MPC, CPO, and PCAP (ours). For SR, higher values are better, for |Fr1 & Rupture values (red%), the lower the better. (Bottom Left) Exp 2: SR for various L--ysstm ternary classes. E.g., the cell 1c:1.r:2, v:98.3) indicates a policy trained on a Ta forest but tested on Tb instance. (Bottom Right) Exp 3: Comparison of SR and Max Joint Torque averaged over trajectories, between PPO & PCAP, Note: ||//| (lower the better) is taken as a proxy for contact impact in real. Rupture values in red show %cases real branches broke the arm had to be hard- stopped due to high forces.
orientation (4D), and the distance of gripper from target (3D). An important exception is that during real deployment, we replace Fill > fu) observation with the classifier prediction and update the reward terms accordingly. 'he results in Fig. top), demonstrate that our PCAP framework is the best suited to avoid plant damage, with a comparable success rate to a baseline that ignores contact impact. To illustrate, the fourth block in both charts, corresponding to the hardest obstruction pattern, indicates that for moderate drop in Success Rate (69 to 59%), the net forces acting on the plant reduce by 3x, while expected plant ruptures reduce by 15x. Additional experimental details, time limits, and success metric definitions can be found in section A.3.
Experiment (L-System Variability): We generate four distinct sets of ternary forests, each cor- responding to classes Ta, Tb, Tc, & Td (Fig. 1). Then, we train PCAP policies on each forest and test its performance on all classes. The result heat map, presented in Fig. bottom left), leads us to the following conclusions. Policies trained class Ta outperform other classes (col 1), whereas the success rate is the lowest with class Te (col 3). While numerous factors may impact the performance of a class, like the stochasticity during the forest generation and the specific combination of occlu- sions and target; primarily. this variation is a result of the hanging arch-like geometry (most evident in class Tc, arising from plant tropism) that varies between classes In contrast, Tb class tasks are the easiest to solve (row 2), for all policies, presumably due to the thinner, pliant outer branches.
Experiment 3 (Real-world Tests): Here, we real tree branches, with varying topology, extracted from two distinct species to obstruct the path between the Kinova arm and reach target. However,
7

---

in real, we compare only two policies: the baseline PPO and our contact-aware policy (PCAP). The policies trained beforehand in the Isaac Gym simulator are deployed in real as checkpoints and are used in conjunction with the pre-trained classifier described in section 3.2. Unlike in simulation, where the contact forces between rigid bodies are available from Isaac Gym, in real, we use the joint torques as a proxy metric to compare PPO and PCAP. Specifically, we compute ||7e|l2 each time step and choose the maximum within each test trajectory indicate plant damage. Overall, the real-world experiment results in (i: bottom right) echo the simulation tests, proving that PCAP mitigates branch impact with solid success rates and successful sim-to-real transfer. Additional experiments are included in the Appendix for time efficiency (A.4), geometry ablation with and without L-system (A.5), dynamics ablation (A.7), and the classifier performance (A.8).
Throughout our experiments, in simulation and real, we observe the agent deploying novel and unexpected strategies evade contacts and reduce contact forces. For lack of better definition, we consider behaviour novel if an ethologist would likely consider it "intelligent" if displayed by a biological organism. We encourage readers to view the extensive videos on our website and the supplementary material to see these behaviours in action.
Discussion:
Some strategies are highly beneficial but unsurprising, such as the arm slowing down (i.e., reducing vobotjoint acceleration) to mitigate the impulse from collision or pushing through regions with lower resistance, like pliant green foliage. In contrast, other demonstrated characteristics are unforeseen. First, the agent learns to use the continuous joints of the Kinova arm at the wrists to perform rolling motion along obstructions. This rolling motion is observed both along and against the gravity axis (i.e., rolling up and down upright limbs) and in lateral directions. Second, from training, the agent forms an abstract awareness of the arch-like canopy shape typical of real-world plants. We deduce this from its attempts to repeatedly favour trajectories through the outer periphery of the arch, where the stiffness is lower, get under the arch where there is resistance, even when shorter path exists through the top. Third, the agent pulls the arm back towards itself, often multiple times, when entangled in runners or facing stiff resistance. Furthermore, we notice the arm sliding on thicker branches, wiggling through occlusions, and shaking off entanglements. Arguably, the most vital result of our framework is the ability to perform reactive exploration of the task space; for instance, the arm pulls back from a heavy impact, changes trajectory direction, and advances again, often in repeated cycles, search of low-cost path. few striking similarities between these emergent behaviours and evolved animal traits nature are listed in Appendix (A.1).
There are multiple scenarios where PCAP fails in both simulation and real (refer to supplement video). In simulation, these are primarily due to the arm failing to reach the target within the specified step limit (sim: 800). Two frequent causes are (i) the arm getting entangled in branches from which cannot retract and (ii) the arm exploring various paths without success. The agent fails the max limit (real: 1000) from exploration in real as well in addition to ruptures from shoving motions and entanglements; although hard-stop in advance in at least some cases.
Failure Modes:
5 Conclusion and Limitations
We introduced Proprioceptive Contact-Aware Policy (PCAP) for the delicate manipulation of tree branches, using simulated forest with realistic branching structures derived from parametric L-system model. Our approach avoids reliance on vision external torque sensors, leveraging pro- prioceptive measurements and a contact detection classifier to train reinforcement learning policies. We demonstrated the effectiveness of our method in both simulated and real-world environments, highlighting its robustness and adaptability to varying dynamics and unseen branch topologies.
Limitations the method include the reliance on simulated environments, which may not capture all real-world complexities, We acknowledge that the [purposeful] absence of vision-based feedback could restrict performance in highly unpredictable scenarios, but the objective was to push the limits of the proprioceptive approach. Future work should explore integrating multi-modal sensory inputs and improving the fidelity of simulated environments to enhance the transferability and efficacy of contact-aware policies in diverse applications.
8

---

Acknowledgments
This work was partially supported the SILVANUS Project through European Commission Funding on the Horizon 2020 call number H2020-LC-GD-2020 Grant Agreement number 101037247. The authors are also grateful to multiple groups at CSIRO, Data61 and the University of Sydney who supported this work.
References
[1] D.-V. Nguyen, L. Kuhnert, and K. D. Kuhnert. Spreading algorithm for efficient vegetation detection in cluttered outdoor environments. Robotics and Autonomous Systems, 60(12): (): 1498- 1507, 2012.
[2] R. Antonova, Yang, Sundaresan, Fox, Ramos, and Bohg. A bayesian treatment of real-to-sim for deformable object manipulation. IEEE Robotics and Automation Letters, 7(3): 5819-5826, 2022.
[3] H. Zhou, X. Wang, W. Au, H. Kang, and C. Chen. Intelligent robots for fruit harvesting: Recent developments and future challenges. Precision Agriculture, 1856 2022.
[4] B. Tang, M. A. Lin, I. Akinola, A. Handa, G. S. Sukhatme, F. Ramos, D. Fox, and Y. Narang- Industreal: Transferring contact-rich assembly tasks from simulation to reality. arXiv preprint wrXv:2305.177110 2023.
[5] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makovicchk,, K. Van Wyk, A. Zhurkevich, B. Sundaralingam, et al. Dextreme: Transfer of agile hn-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robolics and Automation (ICRA), pages 5977-5984. IEEE, 2023.
[6] W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737-744. IEEE, 2020.
[7] Prusinkiewicz and A. Lindenmayer. The algorithmic beauty of plants, chapter pages 58 61. Springer Science & Business Media, 2012.
[8] J. Jacob, Bandyopadhyay, J. Williams, P. Borges, and F. Ramos. Learning to simulate tree- branch dynamics for manipulation. IEEE Robotics and Automation Letters, 2024.
[9] M. Fu, A. Kuntz, R. J. Webster, and R. Alterovitz. Safe motion planning for steerable needles using cost maps automatically extracted from pulmonary images. In 2018 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 1942-4949 IEEE, 2018.
[10] O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[11] Prusinkiewicz. Graphical applications of I-systems. In Proceedings of graphics interface, volume 86, pages 247-253, 1986.
[12] Prusinkiewiczz M. Hammel, J. Hanan, and R. Mech. L-systems: from the theory to visual models of plants. In Proceedings of the 2nd CSIRO Symposium on Computational Challenges in Life Sciences, volume 3, pages 1-32. Citeseer, 1996.
[13] J. Weber and J. Penn. Creation and rendering of realistic trees. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 119-128, 1995.
[14] B. Lintermann and O. Deussen. Interactive modeling of plants. IEEE Computer Graphics and Applications, 19(1):56-65, 1999.

---

[15] A. Runions, B. Lane, and P. Prusinkiewicz. Modeling trees with a space colonization algo- rithm. Nph, 7(63-70):6, 2007.
[16] Y. Zhao and J. Barbič. Interactive authoring of simulation-ready plants. ACM Transactions on Graphics (TOG), 32(4):1-12, 2013.
[17] F. Yandun, A. Silwal, and G. Kantor. Visual 3d reconstruction and dynamic simulation of fruit trees for robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 54 54-55, 2020.
[18] Yandun, T. Parhar, A. Silwal, D. Clifford, Z. Yuan, G. Levine, S. Yaroshenko, and Kantor. Reaching pruning locations vine using deep reinforcement learning policy. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 2400-2406. IEEE, 2021.
[19] C. H. Kim, M. Lee, O. Kroemer, and G. Kantor. Towards robotic tree manipulation: Leveraging graph representations, arXiv preprint arXhv:2311.07479, 2023.
[20] S. Kozlovsky, E. Newman, and M. Zacksenhouse. Reinforcement learning of impedance poli- cies for peg-in- peg-in-hole tasks: Role of asymmetric matrices. IEEE Robotics and Automation Let- ters. 7(4):10898
2022.
[21] A. Jain, M. D. Killpack, A. Edsinger, and C. C. Kemp. Reaching in clutter with whole-arm tactile sensing, The International Journal of Robotics Research, 32(4):458 482, 2013.
[22] T. Pang and R. Tedrake. Easing reliance on collision-free planning with contact-aware con- trol. In 2022 International Conference on Robolics and Automation (ICRA), pages 8375-8381. IEEE, 2022.
[23] H. Honda. Description of the form of trees by the parameters the tree-like body: Effects of the branching angle and the branch length on the shape of the tree-like body. Journal of theoretical biology, 31(2):331-338 1971.
[24] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, Storey, M. Macklin, Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaae gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
[25] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[26] D. Popov, A. Klimchik, and N. Mavridis. Collision detection, localization & classification for industrial robots with joint torque sensors. In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 838-843. IEEE, 2017.
[27] Z. Zhang, K. Qian, B. W. Schuller, and D. Wollherr. An online robot collision detection and identification scheme by supervised learning and bayesian decision theory. IEEE Transactions on Automation Science and Engineering, 18(3):1 14-61556, 2020.
[28] Schulman, Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 177.0:1777.06347,, 2017.
[29] D. Makoviichuk and V. Makoviychuk. rl-games: high-performance framework for rein- forcement learning. Denys88/rLgames, 2022.
[30] Z. Ferguson, M. Li, T. Schneider, Gil-Ureta, T. Langlois, C. Jiang, D. Zorin, D. M. Kaufman, and D. Panozzo. Intersection-free rigid body dynamics. ACM Transactions Graphics, 40 (4), 2021.
10

---

[31] M. Bhardwaj, B. Sundaralingam, A. Mousavian, N. D. Ratliff, D. Fox, F. Ramos, and B. Boots. Storm: An integrated framework for fast joint-space model-predictive control for reactive ma- nipulation. In Conference on Robot Learning, pages 750-759. PMLR, 2022.
[32] C. Friedlander. Thigmokinesis in woodlice. Animal Behaviour, 12(1):164-174, 1964.
[33] D. Treit and M. Fundytus. Thigmotaxis as test for anxiolytic activity rats. Pharmacology Biochemistry and Behavior, 3 (4):9599-9,,, 1988.
[34] H.-.. Lin, G. G. Leisk, and B. Trimmer. Gogbot: caterpillar-inspire soft-bodied rolling robot. Bioinspiration & biomimetics, 2010

---

A Appendix
A.1 Resemblances to Biological Traits
Remarkably, some of the demonstrated PCAP behaviours have evolved in nature as well. For exam- ple, positive thigmokinesiss refers to a pattern in which an organism, such as woodlice [32], slows down in response to external contact. Similarly, many organisms, such as rodents and ground beetles, exhibit both negative and positive thigmotaxis [33], orienting themselves away or towards impact as movement response to tactile stimuli. Likewise, rolling behaviour exhibited by many species for defensive locomotion, such as caterpillars (termed ballistic rolling) [34] and chitons (termed vol- vation). In some instances of our experiments, particularly when rolling and sliding strategies exhibited, we observe that the agent exploits the momentum from the branch resistance to advance the arm in the direction of the opposing force, not unlike squirrels and monkeys which utilise branch momentum to traverse among trees. Nevertheless, we emphasise that given the enormity of evolu- tionary pressures and intents that shape biological complexity, our mention of similarity to animal behaviour should only taken as a curious resemblance rather than as a scientific claim.
A.2 Observation/Rewann Table
The observation and reward combinations for each policy type are presented in Table 1, where 0,, fr. qt. & de represent the joint angle (6D), joint velocity (6D), the gripper orientation (4D), and the distance of gripper from target (3D). The reward parameters Ya, Tax The tes Tp and the scaling factors gd. Is are described section (3.3)
Policy
Ot O,, Or qt, de
Tt
PPO
Id (ra + rbb) 9. T
CPO
0, Gr tt d2
[ga (ra + rb) + Is rs) re + Tp
PCAP (ours)
0,, Oe, qt, dr, 1(11 > fu)
{gg (ra + rb) + Is rz} To + Tp
Table 1: Training observations & rewards provided for each policy
A.3
Experimental Setup
For all simulation experiments, generate 60 random targets within the reach radius of the arm but constrained to the direction of the tree. Collision patterns, numbered to 4 in Fig.(3: top), are constructed either by reorienting a test tree or changing the tree itself to expose the agent to varying occlusion patterns. In any case, the test trees are unseen during the training phase. We define success as the arm's 's end-effector achieving a position within 5ems of the desired target, regardless of the orientation, follows: First, we create an ensemble oracle with 2x PPO, Ix CPO, Ix PCAP, Ix MPC find the feasible tasks within the arm torque, velocity limits based on the intuition that task is solvable if any algorithm finds feasible reach trajectory. Then, we baseline the depicted methods against the oracle with the ideal 100% success. However, for the Contact Force metrics || F2, we average over all the 60 tasks because the branches can rupture in the real world regardless of the reach success. We assume reach targets are known, as fruits and discased regions generally have distinct and consistent visual features (color, texture, shape) that differentiate them from the green of branches and foliage. Even when partially occluded, the conspicuous parts may still provide enough contrast for vision algorithms predict the rest based on learned shapes, which is not the case with branches.
subject to a maximum step limit. The Success Rate (SR) calculated as
The real experimental setup and the comparison metrics are similar to simulation experiments, with a few notable differences. First, we only capture 30 test trajectories real. Second, we hard stop the arm if the branches are likely to rupture to keep the geometry similar between PPO and PCAP. The rupture % values (in red) include both cases where the physical branches broke apart and where the arm was stopped in advance. On the other hand, in simulation, rupture % is computed as the proportion of steps over the trajectory length when the contact forces the branch exceed threshold (c.g., 80N) as a proxy metric. Here, allowed the arm to time out regardless of reach
12

---

failure or extreme forces. Third, for real experiments, due to the limited size of our tests and the sparsity of real branches, we disregard collision-free trajectories that avoid contact between the arm and the tree. In other words, we use only the targets where both PPO and PCAP has at least some interaction with the hindering tree. Finally, we do not use an oracle for real; the SR simply proportion of successful attempted targets.
A.4 Time Efficiency
In every experiment throughout this work, we limit the number of steps (simulation: 800, real: 1000) the agent can take. The experiment in this section compares the normalised count minimum steps the agent takes to reach the target, which we take as a proxy for the clock-time efficiency across all methods. The worst-case normalised reach step count of 1.0 indicates timeout failure while the ideal 0.0 represents success at the first time step. To exemplify, the final block from the results in (4g indicates that for the fifth Collision Pattern, average across 60 tasks, PCAP takes 670 steps to reach the target, while baseline PPO takes 550 steps if the allocated step budget is 1000.
IPD: Baseline R54
CPO: "Cunfact Puralty Only-R33 RS
PCAP: "Contart Pensly "Classifiee-
Normaiised Reach Step Count (RS)
1.0
0.85
0.86
0.65
0.67
0.6
0.54
0.47
0.4
0.0
3
Collision Patterns
Figure 4: Comparison of the number of RL steps taken to reach the target (or timeout) across three policies: PPO, CPO, & PCAP (ours). For RS, lower values are better, indicating lower time taken.
The results demonstrate that, on average, PCAP takes 25% more time steps across collision patterns to reach the target compared to the baseline PPO. We emphasise that this slight loss efficiency in PCAP is the trade-off for increased exploration the task space and additional learnt maneuvers such as rolling and retractions (rather than brute forcing through), resulting in gentler branch con- tacts. We also note that the intermediate CPO method the worst performer primarily due to the significant number of failed tasks from timeouts.
A.5 Geometry Ablation
This section demonstrates the benefits of the novel L- forest generator compared the simpler branching structure proposed in [8]. While is well known that L-system generated topol- ogy generate realistic branching patterns compared naive models (sample images presented in 5ig.5.), show empirical evidence that the richer occlusion patterns are beneficial to robotic applications as well.
In this experiment, we train policies on two independent forests, one generated with the basic design in [8] and the other with our L-system models. In the former, we randomise branch lengths, radius, and divergence angles while ensuring similar span for the part of the tree the robot has access to, whereas for the latter, vary the L-system parameters described in 3.1. The results in Fig. show that, average, the success rate, contact forces, and branch rupture metrics are all superior for
13

---

the L-system geometry. We hypothesise that this is because the recursive fractal-based models allow for variations in geometric parameters, resulting in organic, asymmetrical forms that better capture the irregularities found nature, whereas simple contrived structures, even with noise perturbations, lack this diversity limiting policy robustness.
Figure 5: (Left): A simplistic tree procedurally generated from [8] (Right): An L-System generated tree from the ternary classes Tc. based on our method. Both simulations are presented without any randomisation, highlight the intrinsic advantage of L-system generate diverse occlusion patterns, asymmetric arching angles and non-uniform parent-child splits across the topology.
50
PCAP Simplistic Model SR1
PCAP Simplistic Model
100
PCAP": L-System Model 5R
PCAP": System Model |
86%
Contact Forces [F4. (N)
40
RRupture Tiveshold Breach
Success Rate .SA (36
80
66%
73%
70%
71%
74%
Rupture Threshold - 80.0N
+6%
*5%
60
55%
27.5N
26. 26.5N
+2%
4N
40
16.1N
10%
9.6N
20
10
2
Collision Patterns
Collision Patterns
Figure 6: Ablation study of the geometric models comparing the simplistic branching model from [8] and the L-system model (ours), keeping the Dynamics fixed. The comparison is performed by domain randomising both base models. For SR, higher values are better, for || the lower the better.
A.6 Explicit Classifier vs Raw Joint Torques
As illustrated in Table PCAP uses joint position and velocity measurements along with the classi- fier output as part of the observations o,. We justify excluding raw joint torques 7, with the following arguments: First, real Kinova joint torque measurements and simulation values differ considerably owing to heteroscedastic noise (Fig.8) and inertial parameter discrepancies, posing substantial chal- lenges for zero-shot sim-to-real transfer. The noise is primarily due to indirect torque inference rather than dedicated external sensors [8]. Second, we perform an additional ablation study (Fig. 7), to demonstrate that even with perfect knowledge of the raw torques (in simulation), PCAP has a clear edge over PPO for contact forces (<s 2x) and ruptures (ss 5x) while the success rates are comparable. The results also show that CPO gains ground on the contact force/rupture metrics on adding to or, but sacrificing SR. We posit that the temporal dependency features of the contact detection classifier the primary determinant of PCAP performance, which the RL agent cannot
14

---

access in other instances. However, we acknowledge that swapping the MLP network of PPO for component like RNN may result in performance on par with PCAP. Furthermore, the classifier also uses important non-torque features, such as commanded (due to velocity control mode) and executed joint velocities. It is conceivable that, even without any torque signals, the controller's inability to execute the commanded joint velocities could be an indication the classifier of an obstacle on the way. Third, using explicit classifier allows the operators incorporate expert knowledge apriori into the policy through the dataset collection and the feature selection process; for instance, ensuring contact with all links and in all directions. Finally, this modular architecture provides the flexibility to train and test the contact detector robustness independent of policy learning.
PPO: 'jont
"Jo Torques I--h-
CPO: "Joint Torques- 5R1
CPO: "Joint Torques FF
100
ICnnaaC Penalty "Classifier
PCAP "Contact Peralty "Clussifeer - FiL.
86%
Avg. Contact Forces- 1f-1: (N)
40
-Ruuptue Threshold Breach
Success Rato -R (8)
765%
Rupture Threshold 80.0N
72%
67%
$1.7N
62%
30
60
22% 10%
20
19.3N
72%
17.777.1N
14.7N
20
+5%
<5%
<8%
-5%
Collision Pattems
Collision Pattems
Figure 7: Ablation results comparing PCAP with the explicit classifier PPO and CPO with T included as part of o₁.
Dof-1
Dof-2
Dof-3
3
2
2
Torques
1
Torques
-2
Torques
1
-4
200
400
200
400
0
200
400
Time
Time
Time
Dof-4
Dof.5
D
0.2
0.25
0.3
0.00
WW
0.0
Torques
Torques
0.2
Torques
-0.25
-0.2
0.1
-0.50
0.0
-0.4
-0.75
-0.1
-0.6
200
400
200
400
200
400
Time
Time
Time
Figure 8: A sample prediction given by the classifier. The input in blue is the time series noisy joint torques of the six dofs. Red solid lines represent the ground truth, i.e., the start and end of the obstruction. Yellow dashed lines represent the predicted contact at each time-step.
15

---

A.7
Dynamics Ablation
PCAP: Rudimentary- 5R
PCAP: Rudimentary Ifalz
100
PCAP: Beam Deflection SR1
PCAP: Beam Deflection 1F.b
Ave Contact Forces |F4: (N)
37.
Success Rato -SA (36
80
71%
RRuptuee Threshold Breach
Rupture Threshold - 80.0N
60
*1%
40
+2%
*1% +1%
17.5M
*11%-0..0
11.8N
20
10
7.5N
3
2 Collision Patterss
Collision Patterns
Figure 9: Ablation on the tree dynamics models, keeping the L-system geometry fixed. Comparison of Success Rates (SR) and Average Contact Forces across two dynamics models: Rudimentary and Beam Deflection. For SR, higher values are better, for || the lower the better.
In our final ablation study, we keep the L-system geometry, generated with class Ta, fixed and vary the dynamics between the two models described in section 3.1. From Fig.(9), the superior performance of the Rudimentary model compared to Beam deflection isn't surprising, given that in the former, & is approximately the same for specific branch level between training and test, i.e., except for the additional parameter perturbations while training. This result strongly suggests that integrating system identification for dynamic parameters [8] can significantly boost policy learning.
A.8 Classifier Metrics
Metrics for our contact detection classifier with two off-the-shelf classifiers are given in Fig.(10): 3-Layer Neural Network and a Random Decision Forest. For the real PCAP experiments, we use the latter mainly due to the low false positives.
Random Forest
93.0%
7.0%
Actual
19.69 19.6%
80.4%
True Positive Rate
Neural Network
84.9%
15.1%
Actual
6.4%
93.6%
Random Forest (AUC = 0.95)
Network
0.0
0.2
0.4
0.8
False Positive Rate
Predicted
Figure 10: Confusion matrix & ROC for the two contact detection classifiers
16

---

A.9 L-System Perturbation
We domain randomise L-system morphological parameters Σ by injecting Gaussian Noise while recursing through the production rules. From 11 we observe that low-level perturbations (a 0.01) create structures with limited diversity in topology, identical branches, and similar spans. By contrast, high noise levels (o = 0.25) generate infeasible spans with few prospects for robot engage- ment. We settle for Goldilocks level (o 0.1) with sufficient feasibility but high diversity. Notice the branch radius variations for the corresponding middle row, starting from the trunk. Additionally, we create thousands of tree structures to weed out intractable ones with low contact, move the trees along the gravity axis, and rotate them to maximise occlusions and arm interactions.
0.01
o 0.1
0.25
Figure 11: Effects of perturbations on topology: Samples generated by varying noise levels injected in L-System parameters. (Top Row): too low (a 0.01), (Middle Row): our choice (o 0.1), (Bottom Row): too high (o= 0.25)
17

---

A.10 Hyper Parameters
Param
Parameter Description
Value
E
Young's ' modulus
3e9
gd
Distance reward scaling factor
2.0
Smoothness reward scaling factor
0.01
fu
Simulation force threshold
40N
Simulation rupture threshold
80N
Robot friction
0.0.10001000110010000..011111111
Robot damping
88.80.8..880880..0
Table 2: Additional hyper parameter and configuration settings
18


Reviews:
[{"review_id": "uoG4yyXQkz", "reviewer": "Reviewer_7UhE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors first introduce a method for generating diverse plant simulation models. They then train a reinforcement learning (RL) policy with a contact penalty to reach targets around the plant. Finally, they demonstrate that this policy can be transferred to real-world environments.", "summary_of_recommendation": "While the paper has strong points, particularly in its motivation and technical approach, the major issues related to the clarity of contributions, novelty, and experimental validation need to be addressed. After the rebuttal, the paper quality increases with experiments supporting the claims.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "dot indicates an updated review after the rebuttal.\n\n1. There should be a discussion on the time efficiency of PCAP, i.e. the time it takes to reach the target.\n\n+ The paper has included new experiments on time efficiency. A concern is that the proposed method takes 25% more time compared to the PPO baseline.\n\n2. Please discuss failure cases of PCAP in the results/limitation section. For example, in Figure 3 PCAP has a 25% failure rate (probably 7-8 sequences if the total is 30) on branch pattern 1, and 12% is from branch broken/hard stopped, so what caused the rest of 13% failure cases? \n\n+ The authors discuss major failure cases are time cutoff, the robot is entangled by branches or exploring various paths but failed.\n\n3. Please explain the reason for using the contact classifier and PLAI in PCAP as mentioned earlier.\n\n+ The contact classifier is used for sim2real transfer since simulated joint torques are different from real observations. The authors explained that PLAI is only a minor contribution in the paper.\n\n4. If possible, there could be a comparison with a naive inverse kinematics + impedance control baseline. This baseline will better support the claim that learned RL policy can effectively reach in clutter.\n\n+ The revised manuscript included a new MPC baseline. The MPC baseline has a comparable success rate with the PPO baseline but lower average contact forces.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Je3Fn1ENfi", "reviewer": "Reviewer_fjCp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposed a sim-to-real RL pipeline for a kinova arm with only proprioception to reach targets through tree branches while avoiding excessive contact forces. A branch modeling system is proposed to simulate the structure and compliant behavior of real world trees with a small measurement burden. The trained PPO policy shows notably less average force at the cost of slightly lower success rate, in both simulation and real world.", "summary_of_recommendation": "This work presents some useful novel ideas in a new problem with real world application. The experiments are sufficient to prove the point. However the improvements are at the cost of slowing motion down notably, which is still a problem to be solved before getting closer to the real world application. So I give a weak accept.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* Please comment on the joint torque measurement problem I mentioned above, and update the paper if needed.\n* Please provide a clean definition of success criteria in your paper.\n* Please clarity the feature set introduction section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EoQe206DZA", "reviewer": "Reviewer_DcS1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed to simulate a forest in a procedural way with string dynamics and train a policy to reach point near a tree to reduce contact impact. deploys the agent sim2real directly to the real world.", "summary_of_recommendation": "The paper addresses a significant and impactful problem in robotics, specifically the interaction with and avoidance of tree branches. While the proposed method is sound and demonstrates promising results on a real robot, there are several areas that need improvement for clarity and rigor.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Advantage of the Proposed Simulator Compared to [8]:\n\nPlease provide a detailed explanation highlighting the advantages of the proposed simulator over the method described in [8]. Formally evaluate the proposed simulation's efficiency / accuracy.\n\n- Domain Randomization Choice and Impact:\n\nClarify the logic behind the specific domain randomization choices and include ablation studies for each choice, or cite existing work to justify the choice, to demonstrate the impact of these choices on the performance and robustness of the trained RL agent.\n\n- State Representation:\n\nThe choice made is practical but its implication deserves elaboration. Crucially, the agent observation space contains only the joint pose / velocity, the delta between the current pose and the target and a binary signal of whether it is in collision. The binary classifier is based on a series of joint pose information. Could you address any concerns about the potential violation of the Markov Decision Process (MDP) assumption here. Consider whether a Partially Observable Markov Decision Process (POMDP) would be a more appropriate framework given the current state representation. Further, how does the choice of the state representation help or hinder further application of the proposal?\n\n- Figure 3 Clarification:\n\nImprove the readability of the plots in Figure 3. Ensure the captions are clear and provide sufficient information to understand the data presented. This will help in better conveying the results and their significance.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "22giYLI18A", "reviewer": "Reviewer_2pBB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a RL-based framework to solve a unique task of retrieving an object that is obstructed by flexible tree branches. The authors utilize a procedural model to generate trees and mimic their dynamics in the simulation. This model is then used to provide domain randomization in RL. The authors further propose a contact-aware RL method which takes in proprioceptive signals along with some offline-trained contact classifier. The authors validate their method on real world experiments and showcase emergent behaviors of the method.", "summary_of_recommendation": "I think the paper is a really fun read and has potential impactful applications in agriculture.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How many branches were broken during the experiments? (just curious)\n \n2. What is the intuition for why explicitly building in a contact detector outperforms end-to-end RL with joint torque measurements? If the reward is sufficiently designed than won’t RL eventually learn how to use the joint torque measurements well? (Perhaps it can also directly take in the features like min / max / moment / kurtosis).\n \n3. What is the insight for the smoothness reward $r_s$? Given some trajectories that achieve other rewards equally well, wouldn’t we want to minimize the velocities for smoothness? Might be a sign mistake.\n\n4. How is success exactly defined in Figure 3?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses an important problem of handling trees with branches, and the use of the simulation framework is very helpful. The work can have a good real-world impact on agriculture. However, the reviewers raised concerns about the technical contribution of the paper and how the method improved performance compared to other methods in the past.

----

The paper's quality has been improved notably, and the the authors properly addressed most questions from the reviewers during the rebuttal. The reviewers agree with the authors on the baseline methods now.
Decision: Accept