Gentle Manipulation of Tree Branches: A Contact-Aware Policy Learning Approach

Authors: jayadeep  jacob
shizhe  cai
paulo  borges
fabio  ramos
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 07:00:48

Abstract: Learning to interact with deformable tree branches with minimal damage is challenging due to their intricate geometry and inscrutable dynamics. Furthermore, traditional vision-based modelling systems suffer from implicit occlusions in dense foliage, severely changing lighting conditions, and limited field of view, in addition to having a significant computational burden preventing real-time deployment.In this work, we simulate a procedural forest with realistic, self-similar branching structures derived from a parametric L-system model, actuated with crude spring abstractions, mirroring real-world variations with domain randomisation over the morphological and dynamic attributes. We then train a novel Proprioceptive Contact-Aware Policy (PCAP) for a reach task using reinforcement learning, aided by a whole-arm contact detection classifier and reward engineering, without external vision, tactile, or torque sensing. The agent deploys novel strategies to evade and mitigate contact impact, favouring a reactive exploration of the task space. Finally, we demonstrate that the learned behavioural patterns can be transferred zero-shot from simulation to real, allowing the arm to navigate around real branches with unseen topology and variable occlusions while minimising the contact forces and expected ruptures.

Main paper: 
 introduction: 
 Active interaction with natural deformables, such as tree branches, plants, and grass, has been a persistent challenge in robotics; however, the potential benefits are immense for diverse fields, including agriculture, autonomous navigation, and de-mining. The agriculture industry, in particular, benefits from automation as limited labour resources and excessive wastage threaten sustainable farming practices.Robotic interaction with the natural environment is complex due to various factors. First, imagebased perception, a cornerstone of deformable manipulation in labs, has limited success outdoors [1] , as moving branches, occluding foliage, and poor illumination disrupt camera vision. Second, the complex geometry and non-linear dynamics of natural entities limit model-based methods. Additionally, the deformable behaviour of tree branches necessitates reactive control policies that generalise to unseen scenarios. On the other hand, training model-free learners for long trajectories is computationally expensive, particularly with vision-derived states, while collecting real-world samples is complex. Vision techniques that work well with rigid objects are inadequate for deformable objects [2] like vines and stems. Vision systems also fail to assess rigidity where stress-induced deformation is non-uniform. Consequently, state-of-the-art crop harvesting systems, predominantly vision-based planners, have success rates from 50% to 75% with partial occlusions, dropping to 5% when targets are fully occluded [3] .While collision avoidance is conventional wisdom, contact is inevitable in messy environments; thus, minimising impact cost, both in plant damage and arm torques, is more prudent. Deep reinforcement learning (RL) on powerful GPU-based simulators is a safer, sample-efficient, and costeffective option for contact-rich tasks [4] [5] . However, transferring policies from simulation to reality, known as the sim-to-real problem [6] , is notoriously hard due to discrepancies between the two. Domain Randomisation, referring to introducing perturbations in simulation parameters, helps create robust models generalising well to reality. Effective sim-to-real transfer and path planning with deformable contacts require realistic simulation of intricate geometry and deformation parameters to represent real-world occlusion patterns and interaction behaviour. Given these challenges, this work proposes the following approach: First, we draw inspiration from plant morphology using the parametric L-system (Lindenmayer system) [7] to simulate realistic branching structures coupled with dynamics from crude spring actuation [8] . We domain randomise over both the L-system and dynamic spring parameters to generate a procedural forest for training our Proprioceptive Contact-Aware Policy (PCAP) for a reaching task, targeting, for instance, pruning locations, ripe fruits, or diseased regions. The PCAP states include only observations from proprioceptive sensors, not including vision or external torque sensors in favour of a blind reach policy. An independent classifier predicts contact from internal sensor measurements, which aids reward engineering. Finally, we perform zero-shot sim-to-real transfer by extending techniques from [4] to operate on real branches with varying dynamics. With just thirty minutes of simulation training on an RTX 4090, novel strategies emerge, allowing the arm to explore the task space while minimising contacts and contact impact. These strategies and the reactive behaviour translate well to the real world, where branch topology, obstruction patterns, dynamics parameters, and contact forces are all unknown. To summarise, our contributions are as follows:1) We devise a procedural forest generator based on the L-system formalism and Domain Randomisation, where the individual trees are faithful to the real world in geometric complexity, dimensional accuracy, and occlusion patterns.2) We design an RL-based framework trained in simulation, which derives observations and rewards from an independent collision detection classifier and proprioceptive sensor measurements only.3) We demonstrate that the trained reactive policy and the learned contact minimisation strategies can adapt zero-shot from simulation to real to navigate unseen branch topology and contact patterns with the help of a few engineering optimisations. 
related works: 
 This section reviews advancements in sim-to-real transfer, deformable manipulation and robot-plant interaction. Sim-to-real transfer is essential for applying simulation-trained policies to real-world tasks. Many studies focus on deformable object manipulation and contact-rich tasks. For instance, the IndustReal framework [4] used domain randomisation to bridge the sim-to-real gap. This approach varied simulation parameters, improving policy adaptability and performance in transferring complex assembly tasks. [2] used Bayesian methods and parameter inference to manipulate deformable objects while [9] used sophisticated steering needles to operate inside a deformable environment, human lungs. [6] provided a survey on sim-to-real transfer, categorising methods into domain randomisation, domain adaptation, and imitation learning, highlighting their combined effectiveness. Dextreme [5] and Shadow Hand [10] frameworks focus on agile in-hand manipulation, using high-fidelity simulations and comprehensive RL regimes for successful sim-to-real transfer.Photo-realistic tree representations have been proposed with L-System [7][11] [12], using polygons [13] [14], or with space colonisation algorithm [15] . Furthermore, plant simulations for physical interactions are implemented with expensive FEM models [16] or using cheaper mass-spring systems [17] [8] . From a learning perspective, [18] explored an RL-based system aided by vision for known plant topologies, emphasizing the need for detailed prior knowledge, often impractical in dynamic environments. [19] developed graph representations to predict interactions between robots and plant structures but restricted to simulation where the node positions are known apriori.Policy learning for contact-rich tasks typically relies on external force/torque (F/T) sensors [20] , tactile sensing [10] , or vision [4] . These add-ons work well for contacts localised to the end-effector, but with plant manipulation, contact can occur throughout the arm. Contact-aware reaching in plants using whole-arm tactile sensors is presented in [21] , while internal sensors are used in [22] , but in a model-based indoor setting. In contrast, we use proprioceptive measurements and a model-free context. Accurate models of tree branch dynamics have been proposed [8] , using internal torque sensors to estimate parameters like stiffness and damping. More broadly, [3] surveyed intelligent robots for fruit harvesting, identifying trends and challenges, including the need for advanced perception systems and multi-modal sensors to improve accuracy and reliability in agricultural robotics. 
shape representation & dynamics: 
 L-system: The L-system formalism [7, 23] uses formal grammar theory and fractal geometry to model the morphology of plants and other organic structures like algae. A parallel rewriting process updates morphological attributes by applying production rules, starting from an axiom and operating on a fixed alphabet of symbols. The resulting sequence of strings can be interpreted using turtle graphics [11] , which moves in 2D or 3D space by attributing specific meanings to each symbol, such as move-forward or push-pop from a queue, to generate graphical structures akin to real-world organic growth patterns. The parametric L-system associates parameters with symbols to diversify and control limb generation. Formally, a parametric 0L-system is defined by G = ⟨V, Σ, ω, P ⟩, where V is the alphabet, Σ the parameter set, ω the initial axiom, and P the production rules {p 1 , p 2 , p 3 , ...}. We extend the 'turtle' interpretation to physics simulators by generating independent cylindrical links that can be connected and actuated. We favour ternary branching structures over monopodial and sympodial variations, as real-world manipulation problems are more challenging with the former. We experiment with four ternary structures: classes Ta, Tb, Tc, & Td, from [7] . The ternary L-system we used and the parameters for Tb class Σ b is provided in formulation (1), while the corresponding simulations are shown in Fig. 1(a)-(d) . For detailed rules, parameters, and growth attributes, see [7] ; extensive simulations are available in our website. EQUATIONBranch Dynamics: We extend the coarse-grained abstractions from [8] to model the deformable plant behaviour by approximating branch segments with L-system derived rigid cylindrical links, connected via revolute joints and actuated with proportional-derivative (PD) controllers. However, unlike [8] , where the distribution of stiffness and damping parameters ϕ = {K p , K d } of the torsional mass-spring-damper system (Fig. 2b ) is estimated from real deformations, we do not perform system identification, instead experiment with two models to populate them in advance, namely: a) Rudimentary model: where ϕ is fixed for a given branch level (e.g., R2 in Fig. 2a represents the second level) and it decreases exponentially for each level away from the trunk, starting from an upper bound ϕ u tuned manually based on a real tree branch, and subject to a lower bound ϕ l to avoid simulator instabilities. b) Beam deflection model [19] : where K p = Eπr 4 2l and K d = K p /10, where E is the Young's modulus of the plant material, and (r, l) are the radius and length of the links.We learn a robust policy despite the lack of strong vision priors by randomising over both the shape representation and the dynamics parameters. In the former case, we inject Gaussian perturbations (σ = 0.1) to the L-system parameters Σ, which represents structural traits such as divergence angle (d 1 , d 2 ), elongation rate (l r ), width increase rate (v r ), etc., to create a procedural forest (Fig. 2b ) for each ternary class, where individual instances have unique, diverse, and realistic forms. This paradigm fits neatly with our distributed physics simulator choice Isaac Gym [24] , where individual tree instances can be placed into each parallel environment. As for the dynamics, while the deformation parameters ϕ of the rudimentary model are randomised [25] with a Gaussian (σ = 1.0), we do not perturb the Beam deflection model as the L-system shape randomisation already influences the downstream dynamics. Furthermore, during the policy learning phase in the simulation, we randomise the reach target and the part of the tree the robot has access to.We use proprioceptive measurements to develop a reactive policy without vision guidance, specifically the noisy joint torques, following a supervised learning approach similar to [26][27] . Unlike prior works, however, where local collision detection is the end goal, we incorporate online predictions from a pre-trained model into an RL framework for a reach policy with gentle contact.The proposed online collision detector workflow is as follows: First, we execute arbitrary trajectories with a Kinova 6-DOF Jaco 2 arm, obstructing the path at regular intervals with a soft touch or real branch contact, recording obstruction times. Second, the resulting time series dataset, i.e., sequences of joint velocity θt and joint torque τ t , is annotated with collision labels (1-contact, 0-no contact) for each time step t. Next, we train a binary classifier to detect 'bumps' in the signals, indicating contact. Finally, real-time predictions from the classifier are used for policy learning (section 3.3) as part of the state observation o t and reward computation r t . This classifier is deployed only during RL inference for real-world executions. In contrast, simulation training and testing use a proxy classifier to check if the net rigid body contact force measurements F t ∈ R l×3 , available from Isaac Gym for all l robot links, breach a force threshold, I(∥F t ∥ 2 > f u ). The contact impact threshold f u is manually tuned based on real branch compliance during training.Feature extraction for the classifier focuses on amplitude variations, disregarding the frequency spectrum. We create sliding window features, [τ t , τ t-1 , τ t-2 , ..., τ t-(m-1) ], over the last m time steps. Apart from min and max, we compute statistical moments, including mean, variance, skewness, and kurtosis for each of the six dofs separately. We also use a smoothed version of τ t by averaging the last k neighbours to account for high sensor noise levels. We add joint velocity values θt (commanded and executed) and the raw torque measurements τ t for each joint to the feature set. Overall, we capture 28 trajectories with soft contacts on each of the 7 links of the arm in 4 directions, plus 12 trajectories without obstructions. The final dataset has 90 features, with m = 10 and k = 3 performing best. We tune hyper-parameters manually and use two lightweight classifiers: a 3-layer Neural Network and a Random Decision Forest, settling on the latter for its high specificity.We formulate the reaching task as a discrete-time sequential decision problem where the agent attempts to maximise the expected discounted rewards accumulated by interacting with an environment over T time steps. Traditionally, such problems are modelled as an MDP (Markov Decision Process) framework, defined by the quintuple ⟨S, A, P, R, γ⟩, where S is a finite set of possible states, A the set of actions taken by the agent, P (s ′ |s, a) the state transition model, R(s, a, s ′ ) the reward received on transitioning from s to s ′ upon action a, and γ ∈ [0, 1] the discount ratio, subject to s, s ′ ∈ S and a ∈ A. Given the agent has access only to a noisy representation of the state, o ∈ O, actions can be sampled from a learned stochastic policy π w (a|o) parameterised by the network weights w, i.e., a ∼ π w , where the learning objective was to maximiseE π T -1 t=0 γ t R(s t , a t ). Specifically, we use Proximal Policy Optimisation(PPO) [28] variation from the rl-games library [29] , designed to avoid extensive destabilising updates w.r.t the policy at a previous time step, by maximising a clipped surrogate objective.Action & Observation Space: In our discrete-time setting, each action a t corresponds to a 6-dof velocity target, clamped to the arm limits. In velocity control mode, the arm is essentially noncompliant to external disturbances, generating any amount of force to clear obstacles, subject to joint torque limits. We justify our use of velocity control over the more compliant torque/impedance control with similar reasoning as mentioned earlier, viz. deploying torque/impedance control requires accurate F/T sensor measurements, plant manipulation requires whole body contact detection, and Kinova torque readings suffer from high noise perturbations.On the other hand, our observation o t consists of the robot joint pose, joint velocity, deviation of the end-effector pose from the reach target, end-effector orientation, and most crucially, the predicted contact flag from the aforementioned classifier. Notably, our observation space does not include any real-time inputs from the tree, such as branch images or measurements from visual tags. Besides the cost implications, adding any of these inputs can cause an explosion in training time in the context of a distributed simulator, not to mention the corresponding sim-to-real hurdles.Reward Formulation: Our extensive reward-engineering nudges the arm to evade plant contact, reduce impact upon contact, and minimise ruptures, thereby decreasing overall plant damage while reaching the target. We favour a dense reward function, described in formulation (2), over episodic rewards for faster learning and easier credit assignment during each exploration step.r d = 1 1 + ∥d t ∥ 2 2 2 , r b =    7r d if ∥d t ∥ 2 < 0.0125, 3r d if ∥d t ∥ 2 < 0.025, r d if ∥d t ∥ 2 < 0.05, r s = - 6 j=1 θ2 tj , r c = 1.2 if ∥F t ∥ 2 ≤ f u , 0 otherwise , r p = 0 if ∥F t ∥ 2 ≤ 2f u , -1.2 otherwise. (2)Reward r t at each training time step t consists of five distinct components along with their corresponding scaling factors, which are: a) a distance reward r d that progressively increases as the end-effector reaches closer to the target, where the distance d t ∈ R 3 is computed as the difference of the end-effector position from the target, b) a bonus distance reward r b for close target proximity required for grasps, c) a smoothness reward r s on the six joint velocities, d) a collision reward r c to encourage contact evasion or prefer low impact collisions, and finally, e) a rupture avoidance reward to penalise extreme contact forces. For the proposed PCAP algorithm, we definer t = {g d • (r d + r b ) + g s • r s } • r c + r p ,where g d , g s are scaling factors. The product term with r c ensures that the agent receives distance rewards only for the steps where the contact forces are none or negligible; for instance, contact with leaves or thin limbs. Intuitively, the additional rupture penalty r p is to account for cases where the agent learns to apply quick bouts of extreme forces, in an otherwise smooth path, to rupture branches as a strategic sacrifice to acquire the long-term reward of moving closer to the target through the breach, a behaviour that we experimentally observe.Hardware Setup: Our policies are trained on a server equipped with NVIDIA RTX 4090 GPU, 40-core CPU, and 64GB of RAM, while the Kinova ROS api runs on a second workstation with a basic 8-core CPU, and 32GB of RAM, connected to the robot. They interact with each other via a custom REST service that controls the latency specifications of the arm.The input joint velocity frequency requirement for Kinova ROS API is 60Hz-100Hz due to an onboard DSP that loops close to 10ms. On the other hand, the publish rate for sensor metrics (θ t , θt , q t , e.t.c,) is 10Hz on the ROS topics. Therefore, to prevent oversampling at fetch, we nudge the policies during simulation training to generate actions at 10Hz; moreover, at the time of real deployment, we slice the output action a t into n equal segments to feed the API at a higher frequency (e.g., 60Hz for n = 6), i.e, ∆ θ = a t /n, given ∆ θ ∈ R 6 for the six joints.Segmented Steady State Error Control: Despite manual calibration, discrepancies persist in robot parameters, such as joint damping, friction, and gravity compensation values between simulation and real, and steady-state error is inevitable [4] . Moreover, accurately replicating the complex contact dynamics between the branches and arm is challenging [25] [30], especially when the robot operates in a velocity control mode. As an alternative to parameter estimation, [4] proposes to use Policy Level Action Integrator (PLAI) by integrating policy actions over time, effectively applying them to the desired state s d t in contrast to the current state s t , in other words,s d t+1 = s d t ⊕ a t = s d t ⊕ π(o t ), where ⊕ is the state update operation following a sampled action a t from the policy π. Despite the absence of direct feedback control, except at policy evaluation with the current observation o t , the authors show that PLAI acts as a continuous correction model robust to disturbances, compared to the conventional model, s d t+1 = s t ⊕ π(o t ), that transitions from the current state. However, when the control action regulates joint velocity instead, we observe that the position error accumulates over time due to the integral effect, which is harder to control without direct feedback. Therefore, we follow a middle ground between PLAI and the nominal approach. We update from the last desired state for each segment within a time step but use the current state when updating across the trajectory time steps.EQUATIONIntuitively, formulation (3) could be viewed as local disturbance rejection over the short term, with periodic syncing to the current state to offset accumulated deviations.Robot Parameters: The Domain Randomisation in section (3.1) detailed how we introduce variability in both branch shapes and the tree dynamics. On the other hand, we do not randomise the robot dynamics. Instead, we tune the robot damping and friction parameters heuristically by comparing real torques and velocities to simulation-generated values, whereas the inertial priors are gleaned directly from Kinova specification. Furthermore, we perturb the measured contact forces in simulation with a small Gaussian noise (σ = 1.) to account for classifier generalisation error in real. 
experiments & results: 
 We evaluate the effectiveness of our proposed approach through the following experimental setups:Experiment 1 (Simulation Tests): Here, we compare four distinct policies by ablating over the observations and rewards: First, a baseline PPO that does not include a penalty for high contact forces and can, therefore, apply maximum arm torques (subject to arm torque limits) to push through branch occlusions, disregarding any plant damage. Second, a Contact Penalty Only (CPO) policy uses our reward formulation (2), but that does not take in the classifier predictions towards o t . Third, a Model Predictive Control (MPC) baseline from [31] , however, without a prior for the tree geometry to ensure a fair comparison to our blind model. The fourth is our proposed PCAP approach, which combines both classifier prediction and reward formulation (2) . Specific policy details are presented in Table 1 , where θ t , θt , q t , & d t represent the joint angle (6D), joint velocity (6D), the gripper Max Joint Torques - orientation (4D), and the distance of gripper from target (3D). An important exception is that during real deployment, we replace I(∥F t ∥ 2 > f u ) observation with the classifier prediction and update the reward terms accordingly. The results in Fig. (3: top) , demonstrate that our PCAP framework is the best suited to avoid plant damage, with a comparable success rate to a baseline that ignores contact impact. To illustrate, the fourth block in both charts, corresponding to the hardest obstruction pattern, indicates that for a moderate drop in Success Rate (69 to 59%), the net forces acting on the plant reduce by ≈ 3x, while expected plant ruptures reduce by ≈ 15x. Additional experimental details, time limits, and success metric definitions can be found in section A.3.We generate four distinct sets of ternary forests, each corresponding to classes Ta, Tb, Tc, & Td (Fig. 1 ). Then, we train PCAP policies on each forest and test its performance on all classes. The result heat map, presented in Fig. (3: bottom left), leads us to the following conclusions. Policies trained on class Ta outperform other classes (col 1), whereas the success rate is the lowest with class Tc (col 3). While numerous factors may impact the performance of a class, like the stochasticity during the forest generation and the specific combination of occlusions and target; primarily, this variation is a result of the hanging arch-like geometry (most evident in class Tc, arising from plant tropism) that varies between classes. In contrast, Tb class tasks are the easiest to solve (row 2), for all policies, presumably due to the thinner, pliant outer branches.Experiment 3 (Real-world Tests): Here, we use real tree branches, with varying topology, extracted from two distinct species to obstruct the path between the Kinova arm and a reach target. However, in real, we compare only two policies: the baseline PPO and our contact-aware policy (PCAP). The policies trained beforehand in the Isaac Gym simulator are deployed in real as checkpoints and are used in conjunction with the pre-trained classifier described in section 3.2. Unlike in simulation, where the contact forces between rigid bodies are available from Isaac Gym, in real, we use the joint torques as a proxy metric to compare PPO and PCAP. Specifically, we compute ∥τ t ∥ 2 at each time step and choose the maximum within each test trajectory to indicate plant damage. Overall, the real-world experiment results in Fig. (3: bottom right) echo the simulation tests, proving that PCAP mitigates branch impact with solid success rates and successful sim-to-real transfer. Additional experiments are included in the Appendix for time efficiency (A.4), geometry ablation with and without L-system (A.5), dynamics ablation (A.7), and the classifier performance (A.8).Discussion: Throughout our experiments, in simulation and real, we observe the agent deploying novel and unexpected strategies to evade contacts and reduce contact forces. For lack of a better definition, we consider a behaviour novel if an ethologist would likely consider it "intelligent" if displayed by a biological organism. We encourage readers to view the extensive videos on our website and the supplementary material to see these behaviours in action.Some strategies are highly beneficial but unsurprising, such as the arm slowing down (i.e., reducing robot joint acceleration) to mitigate the impulse from collision or pushing through regions with lower resistance, like pliant green foliage. In contrast, other demonstrated characteristics are unforeseen. First, the agent learns to use the continuous joints of the Kinova arm at the wrists to perform a rolling motion along obstructions. This rolling motion is observed both along and against the gravity axis (i.e., rolling up and down upright limbs) and in lateral directions. Second, from training, the agent forms an abstract awareness of the arch-like canopy shape typical of real-world plants. We deduce this from its attempts to repeatedly favour trajectories through the outer periphery of the arch, where the stiffness is lower, to get under the arch where there is no resistance, even when a shorter path exists through the top. Third, the agent pulls the arm back towards itself, often multiple times, when entangled in runners or facing stiff resistance. Furthermore, we notice the arm sliding on thicker branches, wiggling through occlusions, and shaking off entanglements. Arguably, the most vital result of our framework is the ability to perform reactive exploration of the task space; for instance, the arm pulls back from a heavy impact, changes trajectory direction, and advances again, often in repeated cycles, in search of a low-cost path. A few striking similarities between these emergent behaviours and evolved animal traits in nature are listed in Appendix (A.1).Failure Modes: There are multiple scenarios where PCAP fails in both simulation and real (refer to supplement video). In simulation, these are primarily due to the arm failing to reach the target within the specified step limit (sim: 800). Two frequent causes are (i) the arm getting entangled in branches from which it cannot retract and (ii) the arm exploring various paths without success. The agent fails the max limit (real:1000) from exploration in real as well in addition to ruptures from shoving motions and entanglements; although we hard-stop in advance in at least some cases. 
conclusion and limitations: 
 We introduced a Proprioceptive Contact-Aware Policy (PCAP) for the delicate manipulation of tree branches, using a simulated forest with realistic branching structures derived from a parametric L-system model. Our approach avoids reliance on vision or external torque sensors, leveraging proprioceptive measurements and a contact detection classifier to train reinforcement learning policies. We demonstrated the effectiveness of our method in both simulated and real-world environments, highlighting its robustness and adaptability to varying dynamics and unseen branch topologies.Limitations to the method include the reliance on simulated environments, which may not capture all real-world complexities. We acknowledge that the [purposeful] absence of vision-based feedback could restrict performance in highly unpredictable scenarios, but the objective was to push the limits of the proprioceptive approach. Future work should explore integrating multi-modal sensory inputs and improving the fidelity of simulated environments to enhance the transferability and efficacy of contact-aware policies in diverse applications. 
 
 Appendix: Remarkably, some of the demonstrated PCAP behaviours have evolved in nature as well. For example, positive thigmokinesis refers to a pattern in which an organism, such as woodlice [32] , slows down in response to external contact. Similarly, many organisms, such as rodents and ground beetles, exhibit both negative and positive thigmotaxis [33] , orienting themselves away or towards impact as a movement response to tactile stimuli. Likewise, rolling behaviour is exhibited by many species for defensive locomotion, such as caterpillars (termed ballistic rolling) [34] and chitons (termed volvation). In some instances of our experiments, particularly when rolling and sliding strategies are exhibited, we observe that the agent exploits the momentum from the branch resistance to advance the arm in the direction of the opposing force, not unlike squirrels and monkeys which utilise branch momentum to traverse among trees. Nevertheless, we emphasise that given the enormity of evolutionary pressures and intents that shape biological complexity, our mention of similarity to animal behaviour should only be taken as a curious resemblance rather than as a scientific claim.The observation and reward combinations for each policy type are presented in Table 1 , where θ t , θt , q t , & d t represent the joint angle (6D), joint velocity (6D), the gripper orientation (4D), and the distance of gripper from target (3D). The reward parameters r d , r s , r b , r c , r p and the scaling factors g d , g s are described in section (3.3) Policy o t o t o t r t r t r t PPO θ t , θt , q t , d t g d • (r d + r b ) + g s • r s CPO θ t , θt , q t , d t {g d • (r d + r b ) + g s • r s } • r c + r p PCAP (ours) θ t , θt , q t , d t , I(∥F t ∥ 2 > f u ) {g d • (r d + r b ) + g s • r s } • r c + r pFor all simulation experiments, we generate 60 random targets within the reach radius of the arm but constrained to the direction of the tree. Collision patterns, numbered 1 to 4 in Fig. (3: top) , are constructed either by reorienting a test tree or changing the tree itself to expose the agent to varying occlusion patterns. In any case, the test trees are unseen during the training phase. We define success as the arm's end-effector achieving a position within 5cms of the desired target, regardless of the orientation, but subject to a maximum step limit. The Success Rate (SR) is calculated as follows: First, we create an ensemble oracle with 2x PPO, 1x CPO, 1x PCAP, 1x MPC to find the feasible tasks within the arm torque, velocity limits based on the intuition that a task is solvable if any algorithm finds a feasible reach trajectory. Then, we baseline the depicted methods against the oracle with the ideal 100% success. However, for the Contact Force metrics ∥F t ∥ 2 , we average over all the 60 tasks because the branches can rupture in the real world regardless of the reach success. We assume reach targets are known, as fruits and diseased regions generally have distinct and consistent visual features (color, texture, shape) that differentiate them from the green of branches and foliage. Even when partially occluded, the conspicuous parts may still provide enough contrast for vision algorithms to predict the rest based on learned shapes, which is not the case with branches.The real experimental setup and the comparison metrics are similar to simulation experiments, with a few notable differences. First, we only capture 30 test trajectories in real. Second, we hard stop the arm if the branches are likely to rupture to keep the geometry similar between PPO and PCAP. The rupture % values (in red) include both cases where the physical branches broke apart and where the arm was stopped in advance. On the other hand, in simulation, rupture % is computed as the proportion of steps over the trajectory length when the contact forces on the branch exceed a threshold (e.g., 80N) as a proxy metric. Here, we allowed the arm to time out regardless of reach failure or extreme forces. Third, for real experiments, due to the limited size of our tests and the sparsity of real branches, we disregard collision-free trajectories that avoid contact between the arm and the tree. In other words, we use only the targets where both PPO and PCAP has at least some interaction with the hindering tree. Finally, we do not use an oracle for real; the SR is simply the proportion of successful to attempted targets.In every experiment throughout this work, we limit the number of steps (simulation: 800, real: 1000) the agent can take. The experiment in this section compares the normalised count of minimum steps the agent takes to reach the target, which we take as a proxy for the clock-time efficiency across all methods. The worst-case normalised reach step count of 1.0 indicates a timeout failure while the ideal 0.0 represents a success at the first time step. To exemplify, the final block from the results in Fig. (4) indicates that for the fifth Collision Pattern, on average across 60 tasks, PCAP takes 670 steps to reach the target, while baseline PPO takes 550 steps if the allocated step budget is 1000. The results demonstrate that, on average, PCAP takes 25% more time steps across collision patterns to reach the target compared to the baseline PPO. We emphasise that this slight loss of efficiency in PCAP is the trade-off for increased exploration of the task space and additional learnt maneuvers such as rolling and retractions (rather than brute forcing through), resulting in gentler branch contacts. We also note that the intermediate CPO method is the worst performer primarily due to the significant number of failed tasks from timeouts.This section demonstrates the benefits of the novel L-System-based forest generator compared to the simpler branching structure proposed in [8] . While it is well known that L-system generated topology can generate realistic branching patterns compared to naive models (sample images presented in Fig. 5 .), we show empirical evidence that the richer occlusion patterns are beneficial to robotic applications as well.In this experiment, we train policies on two independent forests, one generated with the basic design in [8] and the other with our L-system models. In the former, we randomise branch lengths, radius, and divergence angles while ensuring a similar span for the part of the tree the robot has access to, whereas for the latter, we vary the L-system parameters as described in 3.1. The results in Fig. 6 show that, on average, the success rate, contact forces, and branch rupture metrics are all superior for the L-system geometry. We hypothesise that this is because the recursive fractal-based models allow for variations in geometric parameters, resulting in organic, asymmetrical forms that better capture the irregularities found in nature, whereas simple contrived structures, even with noise perturbations, lack this diversity limiting its policy robustness. Figure 6 : Ablation study of the geometric models comparing the simplistic branching model from [8] and the L-system model (ours), keeping the Dynamics fixed. The comparison is performed by domain randomising both base models. For SR, higher values are better, for ∥F t ∥ 2 , the lower the better.As illustrated in Table 1 , PCAP uses joint position and velocity measurements along with the classifier output as part of the observations o t . We justify excluding raw joint torques τ t with the following arguments: First, real Kinova joint torque measurements and simulation values differ considerably owing to heteroscedastic noise (Fig. 8 ) and inertial parameter discrepancies, posing substantial challenges for zero-shot sim-to-real transfer. The noise is primarily due to indirect torque inference rather than dedicated external sensors [8] . Second, we perform an additional ablation study (Fig. 7 ), to demonstrate that even with perfect knowledge of the raw torques (in simulation), PCAP has a clear edge over PPO for contact forces (≈ 2x) and ruptures (≈ 5x) while the success rates are comparable. The results also show that CPO gains ground on the contact force/rupture metrics on adding τ t to o t , but sacrificing SR. We posit that the temporal dependency features of the contact detection classifier are the primary determinant of PCAP performance, which the RL agent cannot access in other instances. However, we acknowledge that swapping the MLP network of PPO for a component like RNN may result in performance on par with PCAP. Furthermore, the classifier also uses important non-torque features, such as commanded (due to velocity control mode) and executed joint velocities. It is conceivable that, even without any torque signals, the controller's inability to execute the commanded joint velocities could be an indication to the classifier of an obstacle on the way. Third, using an explicit classifier allows the operators to incorporate expert knowledge apriori into the policy through the dataset collection and the feature selection process; for instance, ensuring contact with all links and in all directions. Finally, this modular architecture provides the flexibility to train and test the contact detector robustness independent of policy learning. 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: This paper addresses an important problem of handling trees with branches, and the use of the simulation framework is very helpful. The work can have a good real-world impact on agriculture. However, the reviewers raised concerns about the technical contribution of the paper and how the method improved performance compared to other methods in the past.

----

The paper's quality has been improved notably, and the the authors properly addressed most questions from the reviewers during the rebuttal. The reviewers agree with the authors on the baseline methods now.

Reviews: review: title: title: The overall method is interesting; however, the current experiments do not sufficiently support the major claims.
paper_summary: paper_summary: The authors first introduce a method for generating diverse plant simulation models. They then train a reinforcement learning (RL) policy with a contact penalty to reach targets around the plant. Finally, they demonstrate that this policy can be transferred to real-world environments.
main_review: main_review: **Strong points**:
The motivation of the paper is sound. Using proprioceptive sensing to navigate cluttered plant environments is compelling due to the complex geometry of plants, which vision alone cannot fully reconstruct. Tactile feedback enables robots to explore compliant contact while avoiding damage to the plants. Additionally, using joint torques from built-in sensors is a good choice, as contacts along the entire arm are reflected in the sensor readings.

The reviewer likes the plant simulation model generation, which is of interest for agriculture robots. Learning RL policy to reach in clutter and using sim2real transfer is a valid approach. The RL-based policy has the potential to learn interesting reaching behaviors from simulation, like the rolling and sliding motions shown in the video. 

**Major complaint**: the key contribution is not clear and novelty in the RL framework is not supported by experiments. 

The authors did not evaluate the time efficiency of the proposed approach. A naive baseline that probes all possible reaching directions with a maximum torque threshold will find a path with a high success rate. The point of using the RL policy is that the robot could reach the goal more efficiently. There does not seem to be a maximum execution time in the experiments.

A key limitation is that the RL policy only uses the current joint angles and torques, and cannot reason geometry in contact to perform global planning. The policy is expected to be trapped when it faces hard branches at a similar location. It is not clear whether the learned policy can be better than the impedance control + retraction heuristic in this case.

Current results do not show enough strength for the proposed approach. The benefit of PCAP over PPO and CPO is expected since the other two have no joint torque signals. Branches occupy a sparse volume and collision may not be frequent for Ta, Tb, and Tc patterns. With the dense Td pattern, the learned policy failed 34% but there is no discussion on failure reasons. 

The novel components in the RL framework are not studied in the experiments:

1. Why not directly use joint torques as observations in the PCAP policy? As shown in Table 1, only classifier output is used in PCAP and no tactile signal is used in the baselines. Concatenating tau_t into o_t should be intuitive. We see the classifier output helps the policy learning a lot, but why a simpler way does not work? Are joint torque readings too noisy to be used directly?

2. There is no ablation on PLAI in results/appendices. If the robot uses velocity control as mentioned in 3.3, the robot dynamics should be sufficiently stiff and the claim of an inaccurate robot dynamics model is not true. The original IndustReal paper used PLAI because of friction and gravity uncertainty, which are indeed difficult to model in simulation.

The related works section should discuss relevant works more systematically.

1. Early works in graphics have used the L-system with biological knowledge. The results in the appendix of the following paper look more photorealistic than the proposed approach:

Jason Weber Joseph Penn, "Creation and Rendering of Realistic Trees," 1995 SIGGRAPH.

But these models are usually polygon soups and need manual editing for simulation, as mentioned in:

Yili Zhao and Jernej Barbič. "Interactive authoring of simulation-ready plants." 2013 TOG. 

To the knowledge of the reviewer, this paper is the first one to generate a diverse plant "simulation" model. So please make this contribution clear in the related works section.

2. Contact-aware reaching in plants has been studied with whole-arm tactile sensors: Jain A, Killpack MD, Edsinger A, Kemp CC. "Reaching in clutter with whole-arm tactile sensing," 2013 IJRR

Reaching using joint torque sensors has been studied in a model-based setting: T. Pang and R. Tedrake, "Easing Reliance on Collision-free Planning with Contact-aware Control," 2022 ICRA

Minor fixes:
1. In the top left part of Figure 3, the success rate should have a max of 100% ticks but currently shows 120%.

2. the authors make broad claims about biology in the last paragraph of the discussion. Please add some references to support the claims.

3. is qt a 4D quaternion?

4. reference 7 is a book and please add section/paragraph locations.
questions: questions: dot indicates an updated review after the rebuttal.

1. There should be a discussion on the time efficiency of PCAP, i.e. the time it takes to reach the target.

+ The paper has included new experiments on time efficiency. A concern is that the proposed method takes 25% more time compared to the PPO baseline.

2. Please discuss failure cases of PCAP in the results/limitation section. For example, in Figure 3 PCAP has a 25% failure rate (probably 7-8 sequences if the total is 30) on branch pattern 1, and 12% is from branch broken/hard stopped, so what caused the rest of 13% failure cases? 

+ The authors discuss major failure cases are time cutoff, the robot is entangled by branches or exploring various paths but failed.

3. Please explain the reason for using the contact classifier and PLAI in PCAP as mentioned earlier.

+ The contact classifier is used for sim2real transfer since simulated joint torques are different from real observations. The authors explained that PLAI is only a minor contribution in the paper.

4. If possible, there could be a comparison with a naive inverse kinematics + impedance control baseline. This baseline will better support the claim that learned RL policy can effectively reach in clutter.

+ The revised manuscript included a new MPC baseline. The MPC baseline has a comparable success rate with the PPO baseline but lower average contact forces.
limitations: limitations: 3
review_summary: review_summary: While the paper has strong points, particularly in its motivation and technical approach, the major issues related to the clarity of contributions, novelty, and experimental validation need to be addressed. After the rebuttal, the paper quality increases with experiments supporting the claims.
title: title: Interesting problem, clean approach, sufficient evaluations
paper_summary: paper_summary: This work proposed a sim-to-real RL pipeline for a kinova arm with only proprioception to reach targets through tree branches while avoiding excessive contact forces. A branch modeling system is proposed to simulate the structure and compliant behavior of real world trees with a small measurement burden. The trained PPO policy shows notably less average force at the cost of slightly lower success rate, in both simulation and real world.
main_review: main_review: The paper has a clean structure and is mostly easy to understand. There are three interesting contributions: one is the efficient modeling of tree branches, making it possible to train a manipulation policy on trees in simulation; The second is to propose a policy structure + reward modeling with a standalone contact detector, whose necessity is demonstrated via ablation studies; the last is to experimentally demonstrate that proprioceptive feedback can be used for force reduction tasks on a robot with no joint torque sensor.

I don’t see major problems in this work, though there are still a few issues need to be fixed:
* One limitation that the paper didn't mention is the missing information from joint torque measurements only. Reading every joint torque does not guarantee the sensing of all contact forces. The further down the contact point is towards the base link, the less force component can be measured. Does your policy exploit this problem and use joint singularity to push branches away?
* It is unclear in the paper how is "success" defined? Does it include any metric on time spent?
* Feature set introduction is confusing. Is \tau the concatenated feature vector or just torque only?

A few comments/suggestions
* Why choosing velocity control over position control? You wouldn’t have the steady state error problem caused by drifting.
* In some of the video, the target is clearly too far away for the robot to reach. The robot has to get close to singularity to meet the goal, at which point there is not much room for different motion solution anyway. Did you setup the goal this way on purpose? In real world applications, you probably want to move the robot base closer to the target to avoid singular poses, which also enables multiple solutions.
* There are robots with better joint torque sensing, e.g. a spot-arm style quasi-direct drive robot, or Franka/iiwa style robot with joint torque sensor. Though it is a contribution to show it even works on a Kinova, I am interested to see how dexterous your policy can be when deployed on a robot better suited for the task.
questions: questions: * Please comment on the joint torque measurement problem I mentioned above, and update the paper if needed.
* Please provide a clean definition of success criteria in your paper.
* Please clarity the feature set introduction section.
limitations: limitations: 2
review_summary: review_summary: This work presents some useful novel ideas in a new problem with real world application. The experiments are sufficient to prove the point. However the improvements are at the cost of slowing motion down notably, which is still a problem to be solved before getting closer to the real world application. So I give a weak accept.
title: title: Sound proposal and promising result but needs improvement for clarity and rigor.
paper_summary: paper_summary: The paper proposed to simulate a forest in a procedural way with string dynamics and train a policy to reach point near a tree to reduce contact impact. deploys the agent sim2real directly to the real world.
main_review: main_review: The paper studies an impactful problem of robotic interaction with tree branches, focusing on collision avoidance. 

The paper propose a method to simulate a forest procedurally using L-system formalism, which is well-documented in existing literature [7][8][13]. They model the dynamics similar to related work [8] and apply domain randomization over the L-system parameters.

The study formalizes a Markov Decision Process (MDP) and trains a reinforcement learning (RL) agent for a reaching task. To minimize collisions, the authors design a binary collision classifier as input to the policy, enabling the agent to be "gentle" with collisions. The trained agent demonstrates a reduced average contact force while maintaining 80-90% of the success rate compared to a baseline that ignores collisions.

Pros:
- The problem is interesting and practically useful.
- The method is sound and demonstrated on a real robot.

Cons:
- The distinction from previous works needs clarification to highlight the novelty.
- The impact of the novel design choices, such as the classifier design and state representation, requires better illustration.
questions: questions: - Advantage of the Proposed Simulator Compared to [8]:

Please provide a detailed explanation highlighting the advantages of the proposed simulator over the method described in [8]. Formally evaluate the proposed simulation's efficiency / accuracy.

- Domain Randomization Choice and Impact:

Clarify the logic behind the specific domain randomization choices and include ablation studies for each choice, or cite existing work to justify the choice, to demonstrate the impact of these choices on the performance and robustness of the trained RL agent.

- State Representation:

The choice made is practical but its implication deserves elaboration. Crucially, the agent observation space contains only the joint pose / velocity, the delta between the current pose and the target and a binary signal of whether it is in collision. The binary classifier is based on a series of joint pose information. Could you address any concerns about the potential violation of the Markov Decision Process (MDP) assumption here. Consider whether a Partially Observable Markov Decision Process (POMDP) would be a more appropriate framework given the current state representation. Further, how does the choice of the state representation help or hinder further application of the proposal?

- Figure 3 Clarification:

Improve the readability of the plots in Figure 3. Ensure the captions are clear and provide sufficient information to understand the data presented. This will help in better conveying the results and their significance.
limitations: limitations: 2
review_summary: review_summary: The paper addresses a significant and impactful problem in robotics, specifically the interaction with and avoidance of tree branches. While the proposed method is sound and demonstrates promising results on a real robot, there are several areas that need improvement for clarity and rigor.
title: title: Review for Gentle Manipulation of Tree Branches: A Contact-Aware Policy Learning Approach
paper_summary: paper_summary: The authors propose a RL-based framework to solve a unique task of retrieving an object that is obstructed by flexible tree branches. The authors utilize a procedural model to generate trees and mimic their dynamics in the simulation. This model is then used to provide domain randomization in RL. The authors further propose a contact-aware RL method which takes in proprioceptive signals along with some offline-trained contact classifier. The authors validate their method on real world experiments and showcase emergent behaviors of the method.
main_review: main_review: <Strengths>

1. The authors tackle quite a challenging problem very creatively, and the results that this task can be done purely from proprioceptive signals stands out as quite surprising. 

2. The emergent behaviors that the authors showcase in the video is also very interesting, and the discussion section over its behavior is also very interesting to read. 

3. I think it’s also good that the authors employ an approach that is actively aware of contact; while other approaches may attempt to do obstacle estimation and collision-free motion planning, the setup of the problem makes both very difficult.   

<Weaknesses>

1. In Figure 3, the baseline PPO seems to have a higher success rate but it’s not mentioned how success is exactly defined / measured.

2. The authors assume that the position of the object-to-grasp is known, but this could use some additional justification. For instance, a vision policy might not be able to detect fine branches and leaves but perhaps it can detect a fruit?
questions: questions: 1. How many branches were broken during the experiments? (just curious)
 
2. What is the intuition for why explicitly building in a contact detector outperforms end-to-end RL with joint torque measurements? If the reward is sufficiently designed than won’t RL eventually learn how to use the joint torque measurements well? (Perhaps it can also directly take in the features like min / max / moment / kurtosis).
 
3. What is the insight for the smoothness reward $r_s$? Given some trajectories that achieve other rewards equally well, wouldn’t we want to minimize the velocities for smoothness? Might be a sign mistake.

4. How is success exactly defined in Figure 3?
limitations: limitations: 3
review_summary: review_summary: I think the paper is a really fun read and has potential impactful applications in agriculture.