Push Past Green: Learning to Look Behind Plant Foliage by Moving It
Authors: Xiaoyu Zhang
Saurabh Gupta
Venue: CoRL 2023 Poster (2023)
Keywords: Deformable Object Manipulation
Model-building
Self-supervision

Abstract: Autonomous agriculture applications (e.g., inspection, phenotyping, plucking fruits) require manipulating the plant foliage to look behind the leaves and the branches. Partial visibility, extreme clutter, thin structures,  and unknown geometry and dynamics for plants make such manipulation challenging. We tackle these challenges through data-driven methods. We use self-supervision to train SRPNet, a neural network that predicts what space is revealed on execution of a candidate action on a given plant. We use SRPNet with the cross-entropy method to predict actions that are effective at revealing space beneath plant foliage. Furthermore, as SRPNet does not just predict how much space is revealed but also where it is revealed, we can execute a sequence of actions that incrementally reveal more and more space beneath the plant foliage. We experiment with a synthetic (vines) and a real plant (Dracaena) on a physical test-bed across 5 settings including 2 settings that test generalization to novel plant configurations. Our experiments reveal the effectiveness of our overall method, PPG, over a competitive hand-crafted exploration method, and the effectiveness of SRPNet over a hand-crafted dynamics model and relevant ablations. Project website with execution videos, code, data, and models: https://sites.google.com/view/pushingfoliage/.

Push Past Green: Learning to Look Behind Plant Foliage by Moving It
Xiaoyu Zhang
Saurabh Gupta
University of Illinois at Urhana-Champaign zhang401@i 1itee000iiiiniiis.. edu
University of Illinois Urbana-Champaign saurabhgeillinois edu
Abstract: Autonomous agriculture applications (e.g., inspection, phenotyping, plucking fruits) require manipulating the plant foliage to look behind the leaves and the branches. Partial visibility, extreme clutter, thin structures, and unknown geometry and dynamics for plants make such manipulation challenging. We tackle these challenges through data-driven methods. We use self-supervision to train SRPNet, a neural network that predicts what space is revealed execution of a candidate action on given plant. We use SRPNet with the cross-entropy method predict actions that are effective at revealing space beneath plant foliage. Fur- thermore, as SRPNet does not just predict how much space is revealed but also where revealed, we can execute a sequence of actions that incrementally re- veal more and more space beneath the plant foliage. We experiment with syn- thetic (vines) and real plant (Dracaena) on a physical test-bed across 5 settings including 2 settings that test generalization to novel plant configurations. Our ex- periments reveal the effectiveness of our overall method, PPG, over a competitive hand-crafted exploration method, and the effectiveness of SRPNet over a hand- crafted dynamics model and relevant ablations. Project website with execution videos, code, data, and models: https://sites.gooleeccoo wow.gle..oo/vve////uusppagaseeee
Keywords: Deformable Object Manipulation, Model-building, Self-supervision
Introduction
The ability to autonomously manipulate plants is crucial in the pursuit of sustainable agricultural practices [1, 2, 3, 4]. Central to autonomous plant manipulation the plant self-occlusion prob- lem. Plants self-occlude themselves (Figure (left).. Plant leaves and branches have to be care- fully moved aside for the simplest of agriculture problems: plant inspection, phenotyping, precision herbicide application, or finding and plucking fruits. This papers tackles this plant self-occlusion problem. We develop methods that learn to manipulate plants so as to look beneath their external foliage. Figure (middle and right) shows steps from sample execution from our method. We believe our work will serve as a building block that enables many different applications that require manipulation of plants in unstructured settings.
Manipulating external plant foliage to reveal occluded space is hard. Sensing is difficult because of dense foliage, thin structures and partial observability. Control and planning is challenging because of unknown dynamics of the plant leaves and branches, and the difficulty of building full articulable plant model. These sensing and control challenges motivate the need for learning. However, use of typical learning paradigms is also not straight-forward. Model-free RL (e.g. PPO [5]) requires interaction data at a scale that is difficult to collect in the real world. Model-based RL is more sample-efficient, but is quite challenging here as precisely predicting the next observation (or state) is hard. Imitation learning is more promising; but for the exploration task tackle, the next best action depends on what has already been explored. This increases the amount of demonstration data required to train models. Lack of high-fidelity plant simulators preclude simulated training.
Our proposal is to tackle this problem through self-supervision [6, 7]. We collect a dataset of action outcomes (amount of space revealed) by letting the robot randomly interact with plants. We use this data train model to predict space revealed by an input action. However, in order to derive a long-term strategy for exploring all of the space beneath the plant, the model has to predict not
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Figure I: (left) Plants self-occlude themselves. Two examples of leaves and branches being pushed aside for inspection and picking fruits. This paper develops learning algorithms that enable robots to tackle this plant self-occlusion problem. We show actions executed by the robot to expose the space behind vines (middle) and Dracaena plant (right).
only how much space would get revealed, but also where (Figure 2 (b), Section 4.1). In this way, the model output lets us reason about what additional space each action would reveal. This allowing us to execute multi-step action sequences that explore all of the area behind the plant using a simple greedy control loop implemented via the cross-entropy method (CEM) (Figure 2 (a), Section 4.3).
This paper implements and tests these ideas on a physical platform that is tasked with revealing space behind decorative vines and a real Dracaena plant. We collect 48 hours of plant interaction data and use it to train neural network that we call Spuce-Revealed Prediction Network (SRPNel). SRPNet, when used with CEM, leads to effective control strategies to reveal all (or user-specified) space beneath the plant foliage. We call our overall framework PushPustGreen (PPG).
Experiments show that SRPNet outperforms a hand-crafted dynamics model and ablated versions of SRPNet. In physical experiments, PPG outperforms a hand-crafted exploration strategy and versions of PPG that replace SRPNet with alternative choices for modeling space revealed. In all 5 settings across vines and Dracaena, including that explicitly test for generalization, we observe relative improvements ranging from 4% to 34% over the next best method. This establishes the benefits of PPG and the use of learning to manipulate plants.
2 Related Work
Autonomous Agriculture. Motivated by the need for adopting sustainable agricultural practices [2,
3, IJ. researchers have sought introduce and expand the use of autonomy for agricultural tasks [8, 91. While full review is beyond our scope, major trends include a) development of specialized robotic hardware [10, 11, 12], b) development of algorithms for perception in cluttered agricultural settings [13, 14, 15], c) design of control algorithms for navigation [16, 17] and manipulation [18), and d) full autonomous farming systems [19, 18, 20, 21].
Plant Manipulation. For manipulation oriented tasks (e.g. fruit picking): [22] compute 3D grasp pose for largely unoccluded fruits, [23] design a visual servoing approach to get partially occluded fruits into full view, [24, 25, 26] output trajectories for reaching fruits while avoiding collisions with plant leaves and branches, and [10, 27] develop soft arms / end-effectors that can maneuver around plant structures. Much less research actually interacts with the plant structure to accomplish tasks. [18] aan--design strategies for pushing fruits out of the way. [28] show simulated results using probabilistic motion primitives for pushing fruits out of the way. We instead study the task of looking behind plant foliage, and hand-crafted strategies proposed in [28, 18] are not directly applicable to our setting. [29, 30] tackle reaching in plants while treating leaves as permeable obstacles, while [31] develops efficient MPC to minimize contact forces when interacting with plants. [32] learn to model object's 's resistance to movement by estimating stiffness distribution from torque measurements. We instead directly model the effect of actions exccuted on the plant.
Manipulation of Deformable Objects. Past works have considered manipulation of other de- formable objects such as cloth [33, 34, 35, 36, 37], ropes [38, 39], elasto-plastics [40, 41], flu-
ids [37, 42, 43], and granular media [44]. [33, 34, 38] design dynamic primitive actions to tackle cloth and rope manipulation. [45, 42, 40, 35] learn particle-based forward models for deformable material and use model--ased RL for control. [46, 41] compose skills for deformable object manip- ulation to solve long-horizon tasks. Our study explores plant manipulation. Lack of high-fideling plant simulators limits the applicability of past methods that rely on large amount of data in simu- lation [42, 36, 38]. At the same time, building dynamics models [40, 35] for plants hard due to dense foliage, thin branch structure, and unknown heterogeneous dynamics.

---

Cross Entropy Method (CEM) with SRPNet for Control
Space revealed by
Rank and
Cross Entropy Method
Execute
Action
Evaluate Sampled Actions
RGB
Height
Space revealed till current step, C
Update space revealed C+11
b) Space Revealed Prediction Network (SRPNet)
Skip connections
Probubility of Revealing Space
Images cropped around action start pixel
Encuder
Decoder
One-hoo action oricntation
One hot height End pixel
Encoder
Figure 2: Overview of PushPastGreen. PushPastGreen learns to manipulate plants to reveal the space behind them thus tackling the plant self-occlusion problem. PushPastGreen includes Space- Revealed Prediction Network (SRPNet) that predicts where space is revealed upon execution of pushing action, shown in (b) and described in Sec. 4.1. SRPNet can not only rank actions based on how much space they will reveal, but because it can also predict where space gets revealed, it can also be used for executing multi-step step trajectories that explore all the space behind the vines as shown in (a) and described in Sec. 4.3. SRPNet is trained using self aaf-supervision as described in Sec. 4.2.
Self-supervised Learning in Robotics. We adopt self-supervised approach for training our mod- els. Self-supervision techniques typically predict scalar quantities (e.g. grasp outcomes [6, 7]. delta cloth coverage workspace [33], pushing+grasping success [47], etc.). Past work has also used self-supervision to build forward models for model-predictive control [48, 49, 50, 51] in pixel or feature spaces. Our work finds middle ground. We predict not just how much space revealed (insufficient for executing sequence of actions), but also where is revealed. This lets us execute sequences of actions that incrementally expose more and more space.
3 Problem Setup
Figure shows the different plants that we tackle, a) decorative vines vertically hanging across board, and real Dracaena plant. The vines involve 2D exploration problem and present challenges due to entanglement, thin structures, and extensive clutter. The real Dracaena plant exhibits a large variation in scene depth leading a 3D problem. The Dracaena plant has big leaves that bend only in specific ways. Thus it requires careful action selection. Both test cases exhibit unknown and heterogeneous dynamics which makes hard manipulate them.
As one can notice in Figures and 3, vines occlude the surface behind the vines. Similarly, the Dra- caena leaves occlude the plant. We refer to this occlusion the plant self-occlusion problem. The task to have manipulation policies that can use the non-prehensile pushing actions (as described below) reveal the space beneath the plant surface.
We use Franka Emika robot and change the end effector to grabber (as also done past work [52, 53]). We use RGB-D cameras pointed at the plant for sensing. Our action space consists of non-prehensile planar pushing actions (also used in past work e.g. [51]). We sample a 3D location and push in a plane parallel to the board for the vines and to the ground for the Dracaena plant. As vines have limited depth variation, we use a fixed for the vines, but actions are sampled at varying for the Dracaena plant. Sections A.I and B. provide more experimental details.
4 Proposed Approach: Push Past Green
PPG adopts a greedy approach. We keep track of space that has not yet been revealed, and execute actions that would reveal the most new space. Doing this requires model that predicts what space

---

View from Azure Kinect
View from Kinect V2
Vines occluding space
Leaves occluding space
Figure 3: Hardware setup for vines (left) and real Dracaena plant (right). We use a grabber as the end-effector [52, 53]. View from the RGB-D camera is in the inset. The task is to move the vines and the Dracaena leaves aside to reveal the space occluded by them.
candidate action would reveal. As plants are complex to model, such a model is hard to hand-craft. Furthermore, is difficult to estimate the precise state and physical parameters for plants from a single RGB-D image (e.g. placement of leaves and branches with respect to another, location and connectivity of all the leaves with the stems, stiffnnes parameters). This precludes the use of physical simulation for such prediction. Thus, design Space-Revealed Prediction Network (SRPNet). that uses learning to directly predict space revealed on execution of a given action on given plant configuration (Section 4.1). Learning directly make this prediction sidesteps the complexity of precise state estimation and physical simulation necessary to build a full dynamics model for the plant. To obtain the data to train SRPNet, we adopt self-supervised approach and execute random actions from the rohot action space. We automatically compute the space revealed after an action using the RGB-D image (Section 4.2). Together with SRPNet, we design PPG, greedy algorithm that uses the cross-entrops method (CEM) [54] to sample the action that promises to reveal the most new space on top of space already revealed (Section 4.3).
4.1 Space- Prediction Network
Input Representation. As shown in Figure (b), the input to our model is a 200 x 200 patch cropped out at the action start location. We crop both the RGB image and an image denoting the height relative to the surface beneath the vines (or relative the ground beneath the Dracaena). The height image is computed using the point cloud from the RGB-D cameras. As the model sees crops around the site of interaction, each action starts the center of the image. We only need to represent the coordinate of the push start location, the push direction (0), and the push distance (d). We represent these using a) a onc-hot vector depicting the push direction, b) a one-hot a height, and c) push distance via the location of the action end-point i.e. [d cos(()) d (f)].
Output. The model produces an output that is the same size as the input. Each value in this spatial map represents the probability that space will get revealed at that location in the image upon execution of input action on the input plant configuration.
Model Archttecture and Loss. We adopt the UNet structure [55] used in image segmentation. The encoder has 5 convolution layers. The action features are processed through 2 transposed convo- lution layers before being concatenated with the visual features and passed to the decoder with transposed convolution layers. We add skip connection between each corresponding convolutional and transposed convolution layer. SRPNet is trained using cross-entropy loss.
4.2 Data Collection and Preparation
Our self-supervised data collection procedure executes random actions from the robot action space. We divide the robot's reachable space into grid of 2em 2em cells. Action starting locations (I., y. 2) are sampled at the centers of these cells. We sample push directions and push by 15cm clipping to the feasible space as necessary. Each interaction executes about 30s. We record RGB- D videos and robot end-effector pose over the entire duration of the interaction. We collected 3529 interactions for vines over hours, and 2175 interactions for Dracaena over 18 hours. We split the dataset into train, val, and test splits in 8:1:1 ratio and train one model for each plant.

---

We automatically compute ground truth for training the model on the collected data. This involves processing the RGB and depth image before and after the interaction. For vines, we found simple decision rule using the color value and change in depth to work well. For Dracaena, very often the entire plant wobbles upon interaction, which leads to erroneous estimates. Thus, we first align the point clouds before and after interaction and then look for depth increase to obtain ground truth. More details are provided Supplementary Sections A.3 and B.3.
4.3 Looking Behind Leaves Using SRPNet
Algorithm describes our control algo-
Algorithm 1: PPG: Revealing space beneath plants.
rithm that uses the trained SRPNet to pre-
dict actions to reveal space behind vines.
Require: Model f that predicts space revealed after action
At each timestep t the trajectory, we
1: Current revealed space, Co + space visible at start
use the cross-entropy method (CEM) [54]
2: for to
do Receive images 7t
to pick out the best action to execute (line
4). We maintain the revealed space far
4:
a, f- CEM J. I+)
(C.). Co is initialized to be the space visi-
5:
Execute action Gt
ble before any actions (line 1). Action pa- rameters are sampled from Gaussian distri-
6:
Calculate additional space revealed
7: Update current revealed space: C441 <- Uee
end for
butions. For each candidate action, SRPNet
predicts where space would be revealed. We determine new space revealed by subtracting the area that has already been revealed (C₂) from SRPNet's output. Samples that are predicted reveal the most new space are selected as clite which are used to fit a Gaussian distribution to sample actions for the next CEM iteration. After all iterations, CEM outputs a, the action found reveal the most new space (line 4). Upon executing we observe the space that is actually revealed and update C₁ (line 6 and 7). The process is repeated for the length of the trial.
Experiments and Results
We test proposed framework through combination of offline evaluations of SRPNet collected dataset (Section 5.1), and online execution on our physical platform for the task of reveal- ing space behind plants (Section 5.2). Our experiments evaluate a) the benefit of learning predict space revealed by actions, b) the effectiveness of SRPNet's input representation, and c) the quality of SRPNet's spatial predictions and selected actions for long-horizon and targeted exploration.
5.1 Offfine Evaluation of SRPNet
We train and evaluate SRPNet on data gathered on our physical setup as de- scribed in Section 4.2. We measure the average precision (AP) for the pix- els labeled as revealed-space. We train on the train split, select model check- points on the validation set, and report performance on the test set Table 1, For vines, we report performance in two settings: Vines [AlI] i.e. seeing the board (behind the vines) counts as revealed space, and Vines [5cm] ie.. height decrease of 5cm counts as re- vealed space. For Dracaena, only see- ing past the leaf (as determined by our automated processing from Section 4.2) counts as revealed space.
Methods
Vines [AlI]
Vines [5em]
Dracaena
Full SRPNet (Our)
46.3
54.4
44,2
Input Representation Ablations
No action
30.2
43,5
28.4
No height map
46.9
49.1
40.6
No RGB
33.4
46.4
28.7
No RGB and no height map
28.4
35.2
10.5
Data Augmentation Ablations
No left/right flips
44.1
52.7
34.6
No color jitter
41.0
51.4
30.7
Average precision for different models at pre- dicting space revealed. Higher is better. Our proposed in- put representation outperforms simpler alternatives and data augmentation boosts performance.
Table 1:
Results. Experiments presented in Table 1 reveal the effectiveness of our method and provide in- sights about the underlying data. First, across all three settings, the full model able to extract information from visual observations to produce higher quality output than just basing the predic- tions on the action information alone (Full model vs. No RGB and no height map). This suggests that plant configuration (as depicted in the visual observations) is important in predicting the action
5

---

outcome. Second, use of action information leads to better predictions (Full model vs. No action). This suggests that different actions at the same site produce different outcomes, and SRPNet is able to make use of the action information to model these differences. Third, looking at the performance across the three settings, both height map and RGB information are useful for accurate predictions. There are no trivial solutions of the form 'space gets revealed where height is high'. Fourth, as we only have a limited number of training samples, data augmentation strategies are effective. Supple- mentary Figure S3 visualizes the predictions from different versions of our model samples from the test set. Note the nuances that our model able to capture in contrast to the ablated versions.
5.2 Online Evaluation for Looking Behind Plants Task
We next measure the effectiveness of our proposed framework (PPG w/ SRPNet) for the task of looking behind plant surface (as introduced in Section 3). We measure the space revealed in units of em2 for vines and number of pixels for Dracaena.
We start out by demonstrating that PPG w/ SRPNet is able to differentiate between good and bad actions. We then demonstrate our method on the task of looking behind plants over 10 time-step horizon. Finally, we tackle the task of revealing space behind a user-specified spatial target. We conduct experiments on both the vines and the Dracaena plant. As plants can't 't be exactly reset, all experiments test generalization to some extent. To further test generalization, we explicitly evaluate performance on plant configurations that differ from those encountered during training.
Inexact resets also pose a challenge when comparing methods. We randomly reset the plant (such as rotating the Dracaena) between trials and expect the variance due to inexact resets to average out over multiple trials. To prevent experimenter or other unknown environmental bias, we a) randomly interleave trials for different methods, and b) reset before revealing which method runs next.
5.2.1 Baselines
Tiling Baseline. For the long-horizon exploration tasks, this hand-crafted baseline randomly sam- ples from action candidates that are spread out across the workspace shown in Figure 4. We aid this baseline by limiting the action candidates to be horizontal pushes for the vines and tan- gential actions for the Dracaena. We found these actions to be more effective than actions in other orientations, see Table $1 for vines and Table S3 for Dracaena.
PPG w/ Other Dynamics Models. To disentangle if the improvement is coming from our learned SRPNet or simply from keeping track of space that has already been revealed (C,) in PPG, we swap SRPNet for other models in PPG. Specifically, we compare to hand-crafted dynamics model (described below) and the SRPNet No Image (i.e. no RGB and no height map) model from Table 1.
We construct hand-crafted forward models for vines and the Dracaena plant. These baseline mod- els represent vines as vertically hanging spaghetti, and the Dracaena plant as 2D radially emanating spaghetti. Figure 5 shows the the induced free space upon action execution for this bascline model.
5.2.2 Results
Single Action Selection Performance. Table 2 compares the effectiveness of PPG w/ SRPNet at picking an action that reveals the most occluded space, against random actions from the robot's 's action space. For the strongest comparison, we limit the random sampling to the most effective actions, horizontal pushes for the vines and tan- gential pushes for the Dracaena, shown in Fig- ure 4. Table reports the average space revealed (along with 95% confidence interval) over at least 20 trials for each method. Our approach leads to a relative improvement of 62% for vines and 49% for Dracaena over this strong baseline. This suggests that our model is able to interpret visual observations identify good interaction sites.
Method
Area Revealed
Vines (cm²)
Dracaena (pixels)
Random Horizontal
211.7 (184222588) 5125.6 (9221.. 74900
Tangential Action
PPG w/ SRPNet (Our)
344.1 (7204)7233 7644.4 (63366 98575)
Table 2: PPG selected actions are more effective at revealing space.
'The criterion to automatically determine revealed space occasionally fails. We manually inspected test runs to confirm and fix the output of the automated method. Note, that this manual inspection is done during evaluation only. No method has access to such manual inspection, neither during training nor during execution.

---

Figure 4: Tiling baseline. Cyan arrows show all action candidates considered, and orange arrows show 10 actions selected during an execution. We aid the baseline by limiting the candidates to the most effective actions (horizontal pushes for vines, tangential pushes for Dracaena).
Figure 5: Hand-crafted dynamics model that represents vines vertically hanging spaghetti (left), and the Dracaena plant as 2D radially em- anating spaghetti (right). Cyan area represents the space revealed by the action (red arrow) un- der this hand-crafted model.
Generalization across Plant Growth Per- formance. To test generalization, we run
June
August
the single action selection experiment (Ta- ble 2) on the Dracaena plant after two- month of growth (in August) but with model trained on data from June. Note the difference between plants in Figure 6 (top). Despite changes in appearance and leaf length, PPG generalizes to the grown Dra- caena and outperforms random tangential action as shown in Figure 6 (bottom).
Method
Area Revealed (pixels)
June
August
Long-horizon Exploration Performance. Next, we study SRPNet can be used for situations that require multiple sequential interactions to reveal space behind plants.
Random Tangential Action 5125.6 0023./114..
6780.7 (4909 8618.1)
PPG w/ SRPNet (Our)
7644.4 (63356 98875) 11551.4 1.4
We evaluate model trained with data col- lected from the June Dracaena on the August Dracaena.
Figure 6:
The task is to maximize the cumulative space revealed over a 10 time-step episode. This further tests the quality of SRPNet which now also needs to accurately predict where it thinks space will be revealed.
We conduct experiments, one on Dracaena and on vines. For the vines, we considered 3 set- tings: a) Base Setting: vine setting used for collecting training data, and novel settings to test generalization: Sparse Vines, and e) Separated Vines. While the last two settings explicitly test generalization, note that the first setting also tests models on novel vine configurations not exactly seen in training. For Dracaena, we only conducted experiments in the Base Setting.
(a)
(b)
(4) Separated Vines (revealable space: cm
(d) Dracaena (revenlable space: 174,380 pixces))
Setting (revealable
Sparse (reveulable space: 2235 cm?)
space:
2735 )w
2000
83k
Space Revealed plixcc))
Spara Rewaa ed
by Actions (cm*))
1500
82k
1000
500
0
10
1.0
10
10
Time Step
Time Step
Time Stop
Step
Tiling
w/ Handcrafted Dynamics
PPG w/ Image
SRPNet
Figure 7: Comparison of different methods for multi-step exploration of space behind plant foliage. We show results in four settings across vines (left) and Dracaena (right). The line plots show average cumulative space revealed by actions up to time step / across 10 trials (along with 95% confidence intervals). SRPNet training data collected in the hase setting shown in (a) for vines and (d) for Dracaena. (b) and (c) are novel settings that test generalization capabilities of our model. Our method (PPG SRPNet) outperforms all baselines (a strong hand-crafted policy, and PPG with other dynamics models) across all settings.

---

Figure plots the average space revealed (in cm² for vines and in pixels for Dracaena) as function of the number of time-steps. We report the mean over 10 trials and also show the 95% confidence interval. Across all three experiments our proposed method achieves the strongest performance. Supplementary Figure $4 and videos on the website show some sample executions.
Results suggests that SRPNet is quite effective at predicting where space will get revealed (PPG w/ SRPNet vs. Tiling). Learning and planning via CEM lets us model complex behavior which is hard to hand-craft. Improvements over the tiling baseline increase as the action space becomes larger (Dracaena vs. vines). Moreover, benefits don't 't just come because of keeping track of revealed space (C₂), but also from the use of SRPNet (PPG w/ SRPNet v.s. PPG w/ Handcrafted Dynamics). Furthermore, our model is able interpret the nuances depicted in the visual information predict good actions (PPG w/ SRPNet vs. PPG w/ SRPNet No Image). SRPNet also leads to benefits in novel vine configurations. Benefits are larger the separated vines case than for the sparse vines. This may be because the separated vines are still locally dense and SRPNet processes local patches.
Targeted Revealing Performance. Our final experiment tackles the task of targeted explo- ration. The task here is to reveal space at a user-defined region, m. Figure (left) shows a sample user-selected region. We tackle this task by setting Co to be m. the complement of the user-defined region. Figure 8 (right) presents the results (same legend as for Figure 7, but with- out Tiling Baseline). Again, as SRPNet reliably models the effect of actions, PPG with SRPNet outperforms PPG with other dynamics models.
Space Revoalee ed
Actions (%6]
so
20-
10
Time Step
Figure 8: PPG is also effective at revealing space
behind a specific spatial target.
Discussion
In this paper, we introduced PPG and SRPNet to tackle the problem of manipulating the external plant foliage to look within the plant (the plant-self occlusion problem). SRPNet uses self-supervisee learning to model what space is revealed upon execution of different actions on plants. This sidesteps the difficulty in perception arising from dense foliage, thin structures, and partial information. PPG derives control strategies using SRPNet via CEM, to output sequence of actions that can incremen- physical platform demonstrate the benefits
tally explore space occluded by plants. Experiments of our proposed learning framework for tackling the plant self-occlusion problem.
7 Limitations
We believe ours is unique and first-of-its-kidd study, but it has its limitations. We note two failure modes. First, PPG sometimes resamples an overly optimistic action (that doesn't 't actually reveal much space, so nothing changes and CEM returns a very similar next action) many times over with- out making progress. Second, each individual push action doesn't 't use visual feedback it can't 't recover from say when a leaf slips from below the gripper. These may mitigated by incorporat- ing spatial diversity while selecting actions and by learning closed loop leaf manipulation policies through imitation. More generally, our overall approach relies on input from RGB-D cameras that known to perform poorly in the wild. This may be mitigated through use of specialized stereo cameras built for farm settings [56]. Our techniques for automatic estimation of revealed space can be improved further using recent point tracking models [57], and it may be useful build models that can predict and keep track of full 3D space. Experiments should be conducted with more diverse real plants. Future work should also rank actions from the perspective of the damage they cause to the plant, perhaps via some tactile sensing [30]. Lastly, while autonomous agriculture provides a path towards sustainable agricultural practices, societal impact of such automation should be studied before deployment.
Acknowledgments
Images in Figure (top) have been taken from YouTube video I and YouTube video 2. This material is based upon work supported by the USDA/NSF AIFARMS National AI Institute (USDA #2020-
8

---

67021-32799). an NSF CAREER Award (IIS-2433733,, an Amazon Research Award, and an NViia Academic Hardware Grant. We thank Matthew Chang and Aditya Prakash for helpful feedback. We thank Kevin Zhang for help setting up robot experiments.
References
[1] Adinor Jose Capellesso, Ademir Antonio Cazella, Abdon Luiz Schmitt Filho, Joshua Farley, and Diego Albino Martins. Economic and environmental impacts of production intensification in agriculture: comparing transgenic, conventional, and agroecological maize crops. Agroe- cology and Sustainable Food Systems, (0003)2215-233) 2016.
[2] H Charles J Godfray and Tara Garnett. Food security and sustainable intensification. Philo- sophical transactions of the Royal Society B: biological sciences, 369(1639)22112273.. 2014.
[3] Jonathan A Foley, Navin Ramankutty, Kate A Brauman, Emily S Cassidy, James S Gerber, Matt Johnston, Nathaniel D Mueller, Christine O' D'Connell, Deepak K Ray, Paul C West, et al. Solutions for a cultivated planet. Nature, 478(7369) 2011.
[4] Aaron M Davis and Jordan Pradolin. Precision herbicide application technologies to decrease herbicide losses in furrow irrigation outflows in a northeastern australian cropping system. Journal of agricultural and food chemistry, 64(20):4021440) 2016.
[5] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrxiv:1707.00347. 2017.
[6] Lerrel Pinto and Abhinav Gupta. Supersizing self-supervisson: Learning to grasp from 50k tries and 700 robot hours. In Proceedings the IEEÉ International Conference Roboties and Automation (ICRA), pages 3406-3413. IEEE, 2016.
[7] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421-464 2018.
[8] Juan Jesús Roldán, Jaime del Cerro, David Garzón-Ramos, Pablo Garcia-Aunon, Mario Garzón, Jorge De León, and Antonio Barrientos. Robots in agriculture: State of art and prac- tical experiences. Service robots, pages 67-90. 2018.
[9] Wouter Bac, Eldert J Van Henten, Jochen Hemming, and Yael Edan. Harvesting robots for high-value crops: State-of-the-art review and challenges ahead. Journal of Field Robotics, 31 (6):888-911, 2014.
[10] Naveen Kumar Uppalapati, Benjamin Walt, Aaron J Havens, Armeen Mahdian, Girish Chowd- hary, and Girish Krishnan. A berry picking robot with a hybrid soft-rigid arm: Design and task space control. In Robotics: Science and Systems, page 95, 2020.
[11] Wyatt McAllister, Denis Osipychev, Adam Davis, and Girish Chowdhary. Agbots: Weeding a field with team of autonomous robots. Computers and Electronics in Agriculture, 163: 104827, 2019.
[12] Abhisesh Silwal, Francisco Yandun, Anjana Nellithimaru, Terry Bates, and George Kan- tor. Bumblebee: A path towards fully autonomous robotic vine pruning. arXiv preprint 12.0092/ 2021.
[13] Harry Freeman, Mohamad Qadri, Abhisesh Silwal, Paul O'Connor, Zachary Rubinstein, Daniel Cooley, and George Kantor. Autonomous apple fruitlet sizing and growth rate tracking using computer vision. arXiv preprint 2212:0::21.005000 2022.
[14] Abhisesh Silwal, Tanvir Parhar, Francisco Yandun, Harjatin Baweja, and George Kantor. A A bust coamination-invariann camera system for agricultural applications. In International Con- ference on Intelligent Robots and Systems, pages 3292-3298. IEEE, 2021.
[15] Francisco Yandun, Abhisesh Silwal, and George Kantor. Visual 3d reconstruction and dynamic simulation of fruit trees for robotic manipulation. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 54-55, 2020.

---

[16] Arun Narenthiran Sivakumar, Sahil Modi, Mateus Valverde Gasparino, Che Ellis, Andres Ba- quero Velasquez, Girish Chowdhary, and Saurabh Gupta. Learned visual navigation for under- canopy agricultural robots. In Robotics: Science and Systems, 2021.
[17] Andres Eduardo Baquero Velasquez. Vitor Akihiro Hisano Higuti, Mateus Valverde Gasparino, Arun Narenthiran Sivakumar, Marcelo Becker, and Girish Chowdhary. Multi-sensor fusion based robust row following for compact agricultural robots. arXiv preprint arXiv:2 /5029, 2021.
[18) Ya Xiong, Yuanyue Ge, Lars Grimstad, and Pál From. An autonomous stawberry-harvestin robot: Design, development, integration, and field evaluation. Journal of Field Robotics, 37 (2):202-224, 2020.
[19] Nicola Strisciuglio, Radim Tylecek, Michael Blaich, Nicolai Petkov, Peter Biber, Jochen Hem- ming, Eldert van Henten, Torsten Sattler, Mare Pollefeys, Theo Gevers, et al. Trimbot2020: an outdoor robot for automatic gardening. In ISR 2018; 50th International Symposium on Robotics, pages 1-6. VDE, 2018.
[20] Mark Presten, Yahav Avigal, Mark Theis, Satvik Sharma, Rishi Parikh, Shrey Aeron, Sandeep Mukherjee, Sebastian Oehme, Simeon Adebola, Walter Teitelbaum, et al. Alphagarden: Learn- ing autonomously tend polyculture garden. arXiv preprint arXiv:2111.06014, 2021.
[21] Soran Parsa, Bappaditya Debnath, Muhammad Arshad Khan, et al. Autonomous strawberry picking robotic system (robofruit). arXiv preprint arXiv:2301.0300 2023.
[22] Hanwen Kang, Hongyu Zhou, and Chao Chen. Visual perception and modeling for au- tonomous apple harvesting. IEEE Access, 8:62151-62163 2020.
[23] Chris Lehnert, Dorian Tsai, Anders Eriksson, and Chris McCool. 3d move to see: Multi- perspective visual servoing for improving object views with semantic segmentation. arXiv preprint arXiv: 07896, 2018.
[24] Lufeng Luo, Hanjin Wen, Qinghua Lu, Haojie Huang, Weilin Chen, Xiangjun Zou, Chenglin Wang, et al. Collision-free path-planning for six-dof serial harvesting robot based on energy optimal and artificial potential field. Complexity, 2018, 2018.
[25] Christoph Schuetz, Joerg Baur, Julian Pfaff, Thomas Buschmann, and Heinz Ulbrich. Evalua- tion of direct optimization method for trajectory planning of a 9-dof redundant fruit-picking manipulator. Proceedings of the IEEE International Conference on Robotics and Automa- tion (ICRA), pages 2660-260 2660. IEEE, 2015.
[26] Alessandra Tafuro, Bappaditya Debnath, Andrea M Zanchettin, and E Amir Ghalamzan. dpmp-deep probabilistic motion planning: A use case in strawberry picking robot. In In- ternational Conference Intelligent Robots and Systems, pages
IEEE, 2022.
[27] Johannes Elfferich, Dimitra Dodou, and Cosimo Della Santina. Soft robotic grippers for crop handling or harvesting: A review. IEEE Access, 10:75428 75443, 2022.
[28) Sariah Mghames, Marc Hanheide, and Amir Ghalamzan. Interactive movement primitives: Planning to push occluding pieces for fruit picking. In International Conference on Intelligent Robots and Systems, pages 2616-2623, IEEE, 2020.
[29] Heramb Nemlekar, Ziang Liu, Suraj Kothawade, Sherdil Niyaz, Barath Raghavan, and Ste- fanos Nikolaidis. Robotic lime picking by considering leaves as permeable obstacles. In International Conference on Intelligent Robots and Systems, pages 3278 72844 IEEE, 2021.
[30] Tapomayukh Bhattacharjee, Phillip M Grice, Ariel Kapusta, Marc D Killpack, Daehyung Park, and Charles C Kemp. robotic system for reaching in dense clutter that integrates model predictive control, learning, haptic mapping, and planning. Georgia Institute of Technology, 2014.
[31] Mare Killpack, Ariel Kapusta, and Charles C Kemp. Model predictive control for fast reaching in clutter. Autonomous Robots, 40:537-560, 2016.
10

---

[32] Shaoxiong Yao and Kris Hauser. Estimating tactile models of heterogeneous deformable ob- jects in real time. In Proceedings of the IEEE International Conference on Robotics and Au- lomation (ICRA), 2023.
[33] Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness dynamic manipulation for cloth unfolding. In Proceedings of the Conference Robot Learning (CoRL), pages 24-33. PMLR, 2022.
[34] Zhenjia Xu, Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Dextairity; Deformable manipulation can be breeze. In Robotics: Science and Systems, 2022.
[35] Zixuan Huang, Xingyu Lin, and David Held. Mesh-based dynamics with occlusion reasoning for cloth manipulation. In Robotics: Science and Systems, 2022.
[36] Thomas Weng, Sujay Man Bajracharya, Yufei Wang, Khush Agrawal, and David Held. Fab- ricflownet: Bimanual cloth manipulation with flow-based policy. In Proceedings of the Con- ference on Robot Learning (CoRL), pages 192-202. PMLR, 2022.
[37] Xingyu Lin, Yufei Wang. Jake Olkin, and David Held. Softgym: Benchmarking deep rein- forcement learning for deformable object manipulation. In Proceedings of the Conference Robot Learning (CoRL), pages 432-448. PMLR, 2021.
[38] Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Itera- tive residual policy: for goal-conditioned dynamic manipulation of deformable objects. In Robotics:
Science and Systems, 2022.
[39] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-basee rope ma- nipulation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 2146-2153. IEEE, 2017
[40] Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning to see, simulate, and shape elasto-plastic objects with graph networks. In Robotics: Science and Systems, 2022.
[41] Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B Tenenbaum, David Held, and Chuang Gan. Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[42] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In Proceedings of the International Conference Learning Representations (ICLR). 2019.
[43] Chau Do, Camilo Gordillo, and Wolfram Burgard. Learning to pour using deep deterministic policy gradients. In International Conference on Intelligent Robots and Systems, pages 3074- 3079. IEEE, 2018.
[44] Connor Schenck, Jonathan Tompson, Sergey Levine, and Dieter Fox. Learning robotic manip- ulation of granular media. In Proceedings of the Conference Robot Learning (CoRL), pages 239-248, PMLR, 2017.
[45] Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In Pro- ceedings of the IEEE International Conference Robotics and Automation (ICRA), pages 1205-1211. IEEE, 2019.
[46] Xingyu Lin, Carl Qi, Yunchu Zhang, Zhiao Huang, Katerina Fragkiadaki, Yunzhu Li, Chuang Gan, and David Held. Planning with spatial-temporal abstraction from point clouds for de- formable object manipulation. Proceedings of the Conference on Robot Learning (CoRI.), 2022.

---

[47] Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee. Alberto Rodriguez, and Thomas Funkhouser. Learning synergies between pushing and grasping with self-supervised deep re- inforcement learning. In International Conference on Intelligent Robots and Systems, pages 4238 4245. IEFE, 2018.
[48] Michael Jordan and David Rumelhart. Forward models: Supervised learning with a distal teacher. Cognitive science, 16(3):377-354 1992.
[49] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: 000668, 2018.
[50] Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interac- tion through video prediction. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[51] Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[52] Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser. Grasping in the wild: Learn- ing 6DOF closed-loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters (RA-L), (33):4978-499 4985, 2020.
[53] Sarah Young, Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy, Proceedings of the Conference on Robot Learning (CoRL), 2020.
[54] Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134:19-67, 2005.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Compuring and Computer-AAsistte Intervention-MICO 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 34-2441.3 Springer, 2015.
[56] Abhisesh Silwal, Tanvir Parhar, Francisco Yandun, Harjatin Baweja, and George Kantor. A bust illumination-invaria camera system for agricultural applications. In International Con- ference on Intelligent Robots and Systems, pages 3292-3298, 2021.
[57] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In Proceedings of the European Conference on Computer Vision (ECCV), pages 59- Springer, 2022.
[58] Kevin Zhang, Mohit Sharma, Jacky Liang, and Oliver Kroemer, modular robotic arm control stack for research: Franka-interface and frankapy. arXiv preprint arXiv:2011.023398, 2020.
12

---

Push Past Green: Learning to Look Behind Plant Foliage by Moving Supplementary Material
A Implementation Details for Vine Experiments
A.1 Robot Action Space
Renson censalu
ipace
(c.1) Stattin umgge
(C Approaaa plast
(2.) Issert ge gible into plant
stow
posoble proding dinetione
(c-4) Prshh
[6.5) Mooe grother bock
(c.83 Retroct graboer
Figure S1: Robot's action space for vine setup. (a) shows the rectified image that we operate in, the region be revealed (red box), and the region that the robot can reach (black box). The robot can execute push actions that start at a pixel (ar, y) in the rectified image and push distance of d an angle 0. We use discrete push directions {0, //6, x x/3, π /2, ...,,T,,,, . } as shown in (b). (c.1) through (c.6) show sample execution of the push action.
The robot's action space consists of non-prehensile pushing actions. As shown in Figure (a), these actions are parameterized by (x, 0, d). Such parameterization for pushing actions has been used past works, egg. [51]. Here, (z, y) denotes
start location for the push interaction on the board, 0 denotes the push angle, and d denotes the push length. As shown in Figure S1 (b), sample to be one of angles from {0, x/G π //33, क 2,, 2nT /3, 5 5//66, πT) We not sample angles greater than because pushing towards the bottom of the vines only drags down the vines and could pull the board over. We assume that the grabber inserts deep enough into the vines to push the vines but not too far knock it over; therefore, the pushes planar actions executed with the value. We estimate the location and orientation of the board and establish coordinate frame that is aligned with the board. Push locations and orientations implement these actions by moving the grabber through 4 waypoints, as shown in Figure S1 (c.2) to Figure S1 (c.5). In Figure S1 (c.4), of the vines. We drive the Franka Emika robot between these waypoints using the Franka-interface and frankapy library [58].
expressed in this coordinate frame. We
the effect of randomly sampled action the state
A.2 SRPNet
For the vine setup, we are unable to position the camera such that it is perpendicular to the board. Therefore, we design SRPNet work rectified images of the scene, such that the camera is looking straight at the vines. This corresponds to using homography to transform the image such that the surface underneath the vines becomes conto-parallel. We build the model to only reason about a 40cm x 40cm neighborhood around the action start location. Parts of the board get occluded behind the robot arm as the robot executes the action. These occluded parts and area with no depth readings are masked out for evaluation and training.
A.3 Data Collection
The robot's actions are the same fronto-parallel plane used for SRPNet as described earlier. We estimate the space that can be safely reached by the robot ahead of time to make sure it is not close joint limits during interactions. The resulting space is roughly 40cm x 40em. We divide the feasible space into 20 x 20 grid. Action starting locations (z, y) sampled at the centers of these
13

---

Push Angle
0
T/3
n/2
2 /
5
π
Full Dataset
Interactions
985
460
360
348
359
433
584
3529
Mean area revealed (em²)
215.7
177.3
93.6
58.8
100.4
180.9
237.1
170.3
Table S1: Statisties for the different push directions in the collected vine dataset. Collected dataset reveals many aspects of the problem. For example, for vines, horizontal push actions (0 and т) are the most effective at this task.
grid squares (ie., 400 possible starting locations). We sample push directions from the 7 possible angles, {0, 7/6, 2x/6..... 6T/6 /6]. and push by 15cm clipping to the feasible space as necessary. Therefore, not all interactions have d = 15; for starting locations near the boundary, d < 15.
Our full dataset contains 3529 interactions (summed roughly 30 hours) collected over 11 different days (nonconsecutive). This data includes 2571 interactions done specifically for the purpose of data collection. The remaining interactions come from when we were developing control algorithms. These don't t follow uniform sampling from the robot's action space and are biased towards horizontal actions since the most effective actions for the baselines are often horizontal actions.
We automatically compute the ground truth for training the model on the collected data. Specifically, we use color thresholding to determine when the surface beneath the vines has been fully exposed. We found this simple strategy to be reasonably robust. Note that while we train and use SRPNet predict whether all vines were moved aside reveal the board, we can process the data other ways to also train the model for other tasks. For example, we can re-purpose the data for a task that involves only looking beneath the first layer of vines. We can re-compute ground truth to identify locations where the height decreased by (say) more than 5em for such a task.
A.4 Cross-entropy Method
Our CEM implementation uses 3 iterations that each evaluate 300 candidate actions. We sample (x, y. 0) from Gaussian distributions. In the first CEM iteration, y, 0 are sampled from Gaussians with different mean and variances, chosen cover the whole action space. The parameters are then discretized to match the distribution from data collection. When sampling actions, we only retain action samples that are feasible (i.e. within the robot's reachable space as shown Figure (a)). Elite samples are the top 20% candidates that have the most amount of new space revealed. Running line 3 to Algorithm (Section 4.3) for vines takes about 5 seconds.
14

---

B Implementation Details for Dracaena Experiments
B.1 Robot Action Space
The robot's action space for Dracaena similar that of vines. However, since the Dracaena leaves are at different heights, we define three possible values that the grabber can insert to. The Dracaena plant body is about 45em tall so we defined the 2 values be about 22.5, 17.5, and 12.5em from the top of the plant. For each & value, planar pushing actions (x, y. 8, d) are defined on plane parallel 3m /4, æ, 5 3r /2, 7 7//4.. The
sample 0 from possible angles: {0, я angles are 45 degrees away from one another instead of 30 degrees used in vines because we want to keep the total number of possible actions reasonable.
to the ground.
revealed
(c.1) Starting image
(c.2) Approuch plant
grabber
(a) Push action in topdown view
(b) Eight possible pushing directions
(c.4) Pash
(c.5) Retract grabher
Figure S2: Dracaena robot action space. Similar Figure S1, (a) shows the image from the
camera, (b) shows the pushing directions, and (c) shows the sample execution of push action.
B.2 SRPNet
Since the Kinect camera is looking down at the Dracaena plant, SRPNet does not work on rectified images as does for vines and instead takes in images from the camera they are. We project action start locations into their image coordinates using the camera intrinsics and crop around the locations obtain local patches input into the network. When training SRPNet, adding another head to predict height decrease in addition to the binary classification head helps AP performance. We use Huber loss with 0.1 to provide an auxiliary loss the network.
B.3 Data Collection
The reachable space of the robot the Dracaena setup roughly 57cm x 53cm and corresponds to 29 x 27 grid of 2cm cells. Similar to the vines" setup, the action starting point (z, y) is sam- pled from these 783 possible locations. Given that pushing from the center of the plant tends to displace it entirely, may sprout. We manually delineate a rectangular region around the plant center and do not sam- ple or execute actions in this region. We also sample from 3 possible values (22.5, 17.5, and 12.5cm from the top of the plant as mentioned before). push directions from possible angles, (0, T T/4, T/ 2, 3m/ 3 x, 5m /4, 3T/ /2, 7 7T//4) /4), and push by 15cm clipping the feasible space essary. Therefore, not all interactions have 15; for starting locations near the boundary, d < 15.
aim to discourage such actions to prevent damage to areas where new leaves
Since the plant wobbles during pushing, we discount the area that is revealed due to whole-plant movement. We construct plant point clouds before and after an action; then, iterative closest point (ICP) is performed to align the two point clouds. During execution, the robot body occludes parts
15

---

Push Angle
0
4/4
x/2
3x/4
5/T/4
3x/2
7t/4
Full Dataset
# Interactions
257
262
295
273
249
297
289
253
2175
Mean revealed (pixels)
1391.4
1138.7
990.4
802.0
1154.0
1110.8
1154.9
1495.2
1147.7
Table S2: Statistics for the different push directions in the collected Dracaena dataset.
of the plant, we mount Intel RealSense camera the wrist fill in these occluded regions to aid ICP. Area where the plant height has decreased in the aligned point cloud is considered to be revealed space.
B.4
Cross-entropy Method
We follow the same algorithm as the one outlined in Algorithm (Section 4.3). The Dracaena CEM uses 3 iterations that each evaluate 300 candidate actions. We sample (I. 0) from uniform distributions within the robot's reachable space. The parameters are then discretized to match the data collection's distribution. Top 20% candidates that reveal the most amount new space chosen as elite samples that are fitted with Gaussian distributions for the next iteration. Running one iteration takes about 7 seconds.
B.5 Comparing Tangential to Random Actions
Method
Area revealed (pixels)
Random Action
3956. 1 [2826.1. 5045.9]
Tangential Action
5125.6 [34133.8, 7147.0]
Table S3: Effectiveness of tangential actions. We execute actions tangent to Dracaena leaves in the Tiling baseline because they reveal more space on average compare to random actions.
We chose horizontal actions for the Tiling baseline of vines because they on average reveal the most amount of space. In order to come up with a similar Tiling baseline for Dracaena, we observe that leaves are pushed aside more easily when the grabber moves tangent the leaves. We verify that tangent actions are better than random actions by comparing average space revealed upon execution of actions from the methods. As shown in Table S3, tangential actions reveal more space than random actions, so we use them the Tiling baseline to test the effectiveness of PPG w/ SRPNet against this strong baseline.
16

---

C Visualizations
(a) RGB Image
(b) Height Image
(c) Ground
(d) SRPNet (No Image)
(e) SRPNet
(f) SRPNet (Full)
Truth
(No Action)
Figure S3: Visualizations of output from our proposed SRPNet. We show examples from test set. The white regions in ground truth images represent space revealed by actions drawn as red arrows. Column (d) shows prediction from SRPNet without image input (i.e. no RGB, no height), column (c) shows prediction from SRPNet without action input, and column (f) shows predictions from SRPNet. The brighter the region, the higher the predicted probability of revealing space. Ground truth revealed space indicates the complexity of the task and suggests why the hand-crafted dynamics model (shown in Figure 5) performs poorly at this task. SRPNet is able to effectively use the visual information to make good predictions.
17

---

t t= 2
t t=1 1
t -=
t= 4
View before Interaction
Executed Action
Space Revealed (Ca+1)
View before Interaction
Space Revealed (Cor)) Executed Action
Figure S4: First five time steps of sample execution from method. Top shows the RGB image before interaction, middle row shows the push action executed, and the bottom row shows the cumulative space revealed far. Our model picks actions that effective revealing space.
18


Reviews:
[{"review_id": "jMjMeNgSl4T", "reviewer": "Reviewer_qgE2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on an autonomous agriculture application. It proposed a learning-based method (SRPNet) to manipulate the plant foliage to look behind the leaves and the branches. The proposed SRPNet predicts revealed space for each action candidate and uses the cross-entropy method to achieve closed-loop control. Finally, the proposed method is evaluated on two plants in the real world.", "summary_of_recommendation": "New application for robot learning and comprehensive evaluation and ablation study on real-world platforms.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "refer to the weakness part in the previous section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### strengths\n1. Autonomous agriculture is an important but less-explored area. This paper proposed a data-driven method to solve an important challenge in this field.\n2. The proposed method is straightforward and easy to understand. The cross-entropy method, simple yet effective, enables multi-step closed-loop control.\n3. The proposed method and comprehensive ablations are evaluated on plants in the real world.\n\n### weaknesses\n1. Action space is not clearly described. For example, how many bins are there for one-hot orientation and z height. Related to this, what's the reason to use one-hot representation for orientation and z height, instead of absolute value?\n2. The task solved in this paper is to maximize the cumulated revealed space. Why this task is important and how does the proposed method help other downstream tasks?\n3. The proposed method is only evaluated on two specific plants. It would be better to conduct a systematic and slightly large-scale evaluation.\n4. plants are deformable and elastoplastic objects. The proposed method doesn't explicitly model the physical properties and dynamics.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "KH8fH3Nk2L", "reviewer": "Reviewer_raTR", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system to predict the gain, in terms of observed space, of a push action on the plant foliage. To this end, SRPNet is proposed to regress the predicted space behind the occlusions as an image, thus providing spatial information. The full system PPG uses the outcome of SRPNet to evaluate sampled actions and decide for the best action that predicts most of the space to be discovered. The networks is trained and tested in two setups, synthetic vines and a real dracaena plant.", "summary_of_recommendation": "Weakness overweight the interesting problem setup.\n\n**After rebuttal**\nI appreciate the effort put into the rebuttal to try answer the open points. \n\nHowever,  the weaknesses I highlighted are not well addressed in my opinion. The system can be reduced to a heuristic search, with no comparisons to a information gain driven (NBV or others) approach. From field robotics deployment, the generalization capabilities are not demonstrated ( it would have been more convincing to try out a different Dracaena in an outdoor environment to show some generalization). And finally, my last point was not about moving a selected leaf, it was about the reasoning to discover, or reveal more in a sequence of decisions given the actions and observations taken so far. But, I understand the confusion from my phrasing.\n\nAll in all, I keep my rating, but as already mentioned I am fine with acceptance if the rest of the reviewers go for it.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- The paper is mostly clearly written and easy to follow.\n- This is an interesting application that has potential.\n- SRPNet follows similar ideas as in other applications (depth completion, semantic scene completion, temporal forecasting, world models) where data-driven methods have shown good predictive results.\n\nWeaknesses:\nThree main concerns raised from this paper:\n1. How to decide for an action it was not totally clear. From algorithm 1 it seems that every time there is a need to sample all possible action (used in training) and then evaluate SRPNet to then take the action that maximizes the information gain. This scales poorly with the action space and the complexity of the plant. Please comment.\n2. The whole PPG idea reads like an ungrounded heuristic. There are plenty of methods for motion planning that could have been adapted to this application, giving the right observation and action spaces, for instance Next Best View (like to maximize discovering), of frontier based ones (where frontiers could be defined as occlusions). Please comment.\n3. The claim of generalization is misleading. For generalization I'm expecting to train in one plant/scenario and test in a different one, maybe same species, in different scenario and environmental conditions. The need for training in the same plant that will be deployed speaks against the usefulness of the approach. Please comment.\n\nFinally, although the problem seems interesting, I guess the real value would be in answering the question \"What is behind this leaf/branch?\" and not just moving everything at once.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "5mRwXSjO5Q", "reviewer": "Reviewer_23Je", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors introduce a robotic method for investigating plants. A large part of that is moving foliage out of the way and the paper presents (a) a neural net that estimates how a certain manipulation would affect visibility and (b) a policy that successively explores the plant, trying to maximize viewing new, previously occluded areas over the course of several manipulations.", "summary_of_recommendation": "This is my favorite paper at this year's CoRL and I'll stand fight to get it accepted. It's very well-written, and presents a straight-forward solution and great experiments for a very difficult problem.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I have a couple more nitpicks and questions, but nothing I'd reject a paper over:\n\n- **Q.1** Less a question/nitpick and more of a recommendation: Maybe it'd be worthwhile investigating an eye-in-hand approach for the robot gripper, i.e. instead of having a static camera, you could build a live 3D reconstruction (e.g. a NeRF) by attaching the camera to the end effector and something like Nvidia's Instant-NGP (https://github.com/NVlabs/instant-ngp, which can create a NeRF in seconds on a recent GPU)\n- **Q.2** You mentioned in your limitations that CEM sometimes retries the same action over and over. Shouldn't CEM update its parameters? I don't fully understand why it'd be doing that.\n- **Q.3** I very much like your Fig. 4 and 5 and the captions for both - in my opinion, good figures are self-explanatory together with their captions and you're doing exactly that. The caption contains a very brief summary and explanation what we're seeing. Along the same \"easy to follow\" lines, I also greatly appreciate Fig.2 because you opted to include example images throughout the diagram that immediately make clear what's going on here without complex formalisms.\n- **Q.4** So your model is pretty plant-type-specific at the moment, right? You train one model for each type of plant because they have very different leaf dynamics, right? Analogous to the spaghetti model, have you considered pre-training your model in simulation? It could be possible to create a large scrunched up sheet of cloth (or virtual spaghetti if you please) in a simulator that's capable of dealing with cloth (like Mujoco) \n- **Q.5** In Tab.1, the performance of. the method without height map is better than with. Do you have any idea why that could be?\n- **Q.6** How did you arrive at 15cm moving distance (line 147)? And how does this affect the risk of damaging the plant? In general, did you see any accidental injury to the plants?\n- **Q.7** This method reminds me a little of DensePhysNet (https://arxiv.org/abs/1906.03853), where a model learns how certain dynamic actions will affect objects and it slowly builds up a model of the physics of every object in the scene. Have you ever considered making this a recurrent model (to track changes over time and explored space better) or would that not be useful in your scenario because sometimes the plant slips out of the end effector?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Strengths\n\n- **S.1** Robotics is already hard and adding plants into the mix is a lot harder. I have huge respect for the authors choosing to tackle such a difficult (and \"messy\") problem and providing real-world experiments.\n- **S.2** The method is clean, easy to follow, easy to implement, yet very useful.\n- **S.3** The paper is well-structured and well-written. All the figures are very illustrative and a lot of attention went into everything. This reads like a very polished paper.\n- **S.4** The experiments illustrate the strength of the contribution well.\n\n### Weaknesses\n\nI honestly can't think of any major concern. My only issue is that the authors use a 2D model for view novelty, not a 3D model but (a) that's not in the scope of this work and (b) the authors clearly state this as future work. The current submission is great as it is.\nI was about to write that I would've liked a video to go along with the submission until I remembered that you referenced a website at the beginning of your paper. I checked out the videos on the website and they're very good.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "-GQ5367XjhI", "reviewer": "Reviewer_N9Cs", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a self-supervision-based approach -- Push Past Green (PPG) -- for learning to de-occlude parts of a plant by sequentially moving plant parts with a (real) robot arm. It also introduces a novel dataset for two different plant types (vines and Dracaena) that consists of ~50 hours of plant interaction and associated ground truth \"space revealed\" metrics (data collected using random robot arm actions). This dataset is used to train the PPG system via self-supervision on the random actions in the dataset, and associated \"space revealed\" scores.\n\nThe model input is an RGB + height image of the plant, a history of space revealed so far, as well as an encoding of the suggested action. A subnetwork called \"Space Revealed Prediction Network\" (SRPNet) takes the action encoding and a crop around the start pixel, encodes this and decodes it into a probability map of space revealed.\n\nExtensive experimental results (using a real robot) showcase the advantages of the method.", "summary_of_recommendation": "The paper is very well-written, proposes a solid approach and dataset for an important task within robo-for-agriculture, and extensively showcases the merits of the proposed method and its various design choices. I think the paper should definitely be accepted. It may however be further improved in various ways (see my comments above), but all-in-all it's a no-brainer for acceptance in my view. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please see my comments / questions under \"WEAKNESSES\" above. Prioritize answering any comments which include a question, e.g. \"Why the use of a CRM approach for action selection? It would be good to see some experimental results that compare some form of pure argmax-style action selection during evaluation.\"\n\nIf space permits, consider also addressing any other comments I have.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**STRENGTHS:**\n\n* **Highly relevant from a robotics-in-agriculture perspective**, and this relevance is well motivated in the main text (e.g. the introduction).\n\n* The suggested approach is well motivated, e.g. Line 29-38 in the introduction. In particular, it motivates the need for the collected dataset. Also, to the best of my knowledge (I'm not an expert on robotics-in-agriculture), the work is put into context of relevant related work (see the related works section) and the suggested approach is well-motivated given the various shortcomings / missed opportunities of past works. To the best of my knowledge, the approach and type of dataset are novel.\n\n* A **relevant, real-data and large-scale dataset has been collected** and will be made **publicly available for future research** within the plant self-occlusion problem.\n\n* **Great that the method can be (and is!) trained in a self-supervised manner**, so that expensive annotations can be avoided.\n\n* **Strengths related to the experimental results:**\n    - **Good set of ablations** that showcase the design choices (see e.g. Table 1). The associated analyses (see e.g. Line 199-211) are good too, and provide interesting insights (e.g. \"There are no trivial solutions of the form 'space gets revealed where height is high'.\").\n    - Good that **multiple runs with different random seeds are done**, and that confidence intervals are included.\n    - Good that **runtimes are reported** (albeit in the supplementary material; such an important metric may be best to add to the main paper).\n    - The **proposed method outperforms reasonable baselines.**\n\n* **Very impressive additional material**, including the video summary on the attached webpage.\n\n**WEAKNESSES:**\n\n* **Some things were not quite clear to me and would improve the paper if addressed:**\n    - **Why the use of a CRM approach for action selection?** It would be good to see some experimental results that compare some form of pure argmax-style action selection during evaluation.\n    - In Table 2, I interpreted the \"Random Horizontal / Tangential Action\" as equal to the \"Tiling Baseline\". If this is true, consider renaming \"Random Horizontal / Tangential Action\" to \"Tiling Baseline\". If not, consider clarifying.\n    - It's sometimes hard to grasp the Area Revealed metric when it is reported e.g. in cm^2. Could one perhaps also include some metric which is in the [0,1]-range, i.e. normalized somehow?\n    - It was a bit difficult for me to understand the difference / connection between \"PPG w/ Other Dynamics Models\" and \"Hand-crafted Dynamics Model\". It seems like the latter could be a subset of the former (?).\n    - In Fig. 7, I was a bit confused why the methods require multiple steps to reveal a single plant part? It seems a single, deterministic step would do it, given that the targeted area is quite small (?).\n\n* The greedy nature of the reveal actions imply that **the proposed methodology may be inherently ill-suited for certain types of inferences.** The greedy reveal actions are certainly good for the specific task of revealing things (assuming all reveals are considered equally important / informative -- see my next remark in the below item), but if the revealing is merely a step towards some form of 'global-level' inference related to the plant (e.g. its overall health status), then greedy reveals may not be the globally optimal approach for said global inference. An RL-based method with a sparse reward (obtained at the end) could be a better choice in such cases, but on the other hand it is well-argued why such an approach is difficult in these contexts (see e.g. the second paragraph of the intro).\n\n* Oftentimes, **not all visible space is equally important for a given type of inference** (it may e.g. be that some parts of a plant typically does not hide anything 'interesting', and so more focus should be put on revealing those parts that are likely to hide something interesting (e.g. berries)). It would have been great to see some discussion (e.g. in the Limitations section) that mentions this fact, and ideas for addressing it. \n\n* In the same spirit as the last item above: **It would have been great to have some form of experimental results on downstream performance** (e.g. the task of counting things that are occluded by plant parts (e.g. berries)), and not only evaluate the \"reveal\" metric for which the model is explicitly trained. I however understand that the scope of a paper has its limits.\n\n* While there is quite some degree of out-of-domain generalization being showcased, some quite simple tricks could have further assessed this. For example, the data collection could have been done in different lightning conditions, so that models trained in a certain lightning condition would be tested on another condition.\n\n* Again, great with seed sensitivity analyses. But since the number of trials is less than 20 (it's 10 as I understood it), perhaps +- std would have been sufficient, rather than 95% intervals (since they are not statistically significant given the few trials (less than 20), if I don't misremember my statistics-101-class).\n\n**SUGGESTIONS (not part of evaluation):**\n\n* Might be nice with a last column in Table 1, that simply computes the average numbers across the three existing columns.\n\n* Might be interesting future work to combine your proposed methodology with some form of methods that predict what lies beneath plant parts (similar to e.g. human pose estimation, where occluded body parts are often estimated, albeit at varying accuracy). The policy may then adapt is \"reveal movements\" based on how accurate it thinks that the model already is at predicting what is beneath the plant part even before the reveal. In other words, only those plant parts are revealed where the model cannot confidently guess what is beneath.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on manipulating plants to reveal hidden areas (e.g. moving a leaf). The method predicts revealed space for a given action and uses CEM to move the plants to reveal as much space as possible. Evaluations are conducted on two real-world plants.

Strengths:
- The presentation is clear and readable
- The method is sound and makes sense for this application
- The results demonstrate that the method is effective at revealing space behind plant parts

Weaknesses:
- It is not clear that what the method does is useful for agricultural (or other applications). Simply revealing occluded space does not seem like the correct metric. Rather, revealing specific targets (e.g. fruit or side-shoots) is what such manipulation should seek to do. As the method does not have knowledge of the overall plant structure, it seems the only way to find specific targets is to reveal as much space as possible, which is very inefficient.
- The generalization claim is overstated. It is expected that such a method transfer across a species of plants, but instead the training is plant-specific, requiring that data is collected on the specific plant to be manipulated. This is clearly not going to scale to agricultural applications. The authors should evaluate how transferable the method is across different plants. The claims about the plant growing over the time the research was conducted are not quantified and difficult to evaluate for generalization.

Overall, the rebuttal did not significantly change the reviewers' opinions.
Decision: Accept (Poster)