Three Dimensional Reconstruction of Botanical Trees with Simulatable Geometry
Authors: Ed Quigley
Winnie Lin
Yilin Zhu
Ronald Fedkiw
Venue:  (2021)

Abstract: We tackle the challenging problem of creating full and accurate three dimensional reconstructions of botanical trees with the topological and geometric accuracy required for subsequent physical simulation, e.g. in response to wind forces. Although certain aspects of our approach would benefit from various improvements, our results exceed the state of the art especially in geometric and topological complexity and accuracy. Starting with two dimensional RGB image data acquired from cameras attached to drones, we create point clouds, textured triangle meshes, and a simulatable and skinned cylindrical articulated rigid body model. We discuss the pros and cons of each step of our pipeline, and in order to stimulate future research we make the raw and processed data from every step of the pipeline as well as the final geometric reconstructions publicly available.

Under review as a conference paper at ICLR 2021
THREE DIMENSIONAL RECONSTRUCTION OF BOTANICAL TREES WITH SIMULATABLE GEOMETRY
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle the challenging problem of creating full and accurate three dimensional reconstructions of botanical trees with the topological and geometric accuracy required for subsequent physical simulation, e.g. in response to wind forces. Al- though certain aspects of our approach would benefit from various improvements, our results exceed the state of the art especially in geometric and topological com- plexity and accuracy. Starting with two dimensional RGB image data acquired from cameras attached to drones, we create point clouds, textured triangle meshes, and simulatable and skinned cylindrical articulated rigid body model. We dis- cuss the pros and cons of each step of our pipeline, and in order to stimulate future research we make the raw and processed data from every step of the pipeline well as the final geometric reconstructions publicly available.
INTRODUCTION
fluman-inhabilg outdoor environments typically contain ground surfaces such grass and roads, transportation vehicles such as cars and bikes, buildings and structures, and humans themselves. but are also typically intentionally populated by large number of trees and shrubbery; most of the motion in such environments comes from humans, their vehicles, and wind-driven plants/trees. Tree reconstruction and simulation are obviously useful for AR/VR, architectural design and modeling, film special effects, etc. For example, when filming actors running through trees, one would like to create virtual versions of those trees with which a chasing dinosaur could interact. Other uses include studying roots and plants for agriculture (Zheng et al., 2011; Estrada et al., 2015; Fuentes et al., 2017) or assessing the health of trees especially in remote locations (similar in spirit to Zuffi et al. (2018)). 2.5D) data, i.e. 2D images with some depth information, is typically sufficient for robotic navigation, etc.; however, there are many problems that require true 3D scene understanding to the extent one could 3D print objects and have accurate geodesics. Whereas navigating around objects might readily generalize into categories or strategies such as 'move left,' 'move right,' 'step up.' "go under,' etc., the 3D object understanding required for picking up cup, knocking down a building, moving a stack of bricks or a pile of dirt, or simulating a tree moving in the wind requires significantly higher fidelity. As opposed to random trial and error, humans often use mental simulations to better complete a task, e.g. consider stacking a card tower, avoiding a falling object, or hitting a baseball (visualization is quite important in sports); thus, physical simulation can play an important role in end-to-end tasks, e.g. see Kloss et al. (2017); Peng et al. (2017); Jiang & Liu (2018) for examples of combining simulation and learning.
Accurate 3D shape reconstruction is still quite challenging. Recently, Malik argued! that one should not apply general purpose reconstruction algorithms to say car and tree and expect both recon- structions to be of high quality. Rather, he said that one should use domain-specinf knowledge as he has done for example to rely on the prior that many indoor surfaces are planar in order to reconstruct office spaces (Huang et al., 2017) or entire buildings (Armeni et al., 2016; 2017). Along the same lines, Zuffi et al. (2018) uses base animal shape prior for their reconstructions of wild animals. Thus, similarly take a specialized approach using a generalized cylinder prior for both large and medium scale features.
Kanazawa et al. (2018). Another example of this specialization strategy is
In Section 3, we discuss our constraints data collection well the logistics behind the choices we made for the hardware (cameras and drones) and software (structure from motion, multi-view
Jitendra Malik, Stanford cn231n guest lecture, 29 May 2018

---

Under review as a conference paper at ICLR 2021
stereo, inverse rendering, etc.) used to obtain raw and processed data. Section 4 discusses our use of machine learning, and Section 5 presents a number of experimental results. In Appendices A, B, and C we describe how create geometry from the data with enough efficacy for physical simulation.
2 PREVIOUS WORK
Tree Modeling and Reconstruction: Researchers in computer graphics have been interested in modeling trees and plants for decades (Lindenmayer, 1968; Bloomenthal, 1985; Weber & Penn, 1995; Prusinkiewicz et al., 1997; Stava et al., 2014). SpeedTree2 is probably the most popular software utilized, and their group has begun to consider the incorporation of data-driven methods. Amongst the data-driven approaches, Tan et al. (2007) is most similar to ours combining point cloud and image segmentation data to build coarse-scale details of a tree; however, they generate fine-scale details procedurally using a self-similarity assumption and image-space growth constraints, whereas we aim to capture more accurate finer structures from the image data. Other data-driven approaches include Livny et al. (2010) which automatically estimates skeletal structure of trees from point cloud data, Xie et al. (2015) which builds tree models by assembling pieces from a database of scanned tree parts, etc.
Many of these specialized, data-driven approaches for trees are built upon more general techniques such as the traditional combination of structure from motion (see e.g. Wu (2013)) and multi-view stereo (see e.g. Furukawa & Ponce (2010)). In the past, researchers studying 3D reconstruction have engineered general approaches to reconstruct fine details of small objects captured by sen- sors in highly controlled environments (Seitz et al., 2006). At the other end of the spectrum, searchers have developed approaches for reconstructing building- or even city-scale objects using large amounts of image data available online (Agarwal et al., 2009). Our goal is to obtain 3D model of a tree with elements from both of these approaches: the scale of a large structure with the fine details of its many branches and twigs. However, unlike in general reconstruction approaches, we cannot simply collect images online or capture data using a high-end camera.
To address similar challenges in specialized cases, researchers take advantage of domain-specifer prior knowledge. Zhou et al. (2008) uses a generalized cylinder prior (similar to us) for recon- structing tubular structures observed during medical procedures and illustrates that this approach performs better than simple structure from motion. The process of creating a mesh that faithfully reflects topology and subsequently refining its geometry is similar in spirit to Xu et al. (2018), which poses a human model first via its skeleton and then by applying fine-scale deformations.
So far, our use of networks is limited to segmentation tasks, where we rely on segmentation masks for semi-automated tree branch labeling. Due to difficulties in getting sharp details from convolutional networks, the study of network-based segmentation of thin struc- tures is still an active field in itself; there has been recent work on designing specialized multiscale architectures (Ronneberger et al., 2015; Lin et al., 2017: Qu et al., 2018) and also on incorporating perceptual losses (Johnson et al., 2016) during network training (Mosinska et al., 2018).
Learning and Networks:
RAW AND PROCESSED DATA
As a case study, we select a California oak (quercus agrifolia) as our subject for tree reconstruction and simulation (see Figure 1). The mere size of this tree imposes number of restrictions our data capture: one has to deal with an outdoor, unconstrained environment, wind and branch motion will be an issue, it will be quite difficult to observe higher up portions of the tree especially at close proximities, there will be an immense number of occluded regions because of the large number of branches that one cannot see from any feasible viewpoint, etc.
In an outdoor setting, commodity structured light sensors that use infrared light (e.g. the Kinect) fail to produce reliable depth maps as their projected pattern is washed out by sunlight; thus, we opted to use standard RGB cameras. Because we want good coverage of the tree, we cannot simply capture images from the ground; instead, we mounted our cameras on a quadcopter drone that was piloted around the tree. The decision use drone introduces additional constraints: the cameras must be
2

---

Under review as a conference paper at ICLR 2021
Figure I:
We target California oak for reconstruction and simulation. (Inset) The drone and camera setup used to collect video data of the tree.
lightweight, the camera locations cannot be known a priori, the drone creates its own air currents which can affect the tree's 's motion, etc. Balancing the weight constraint with the benefits of using cameras with a global shutter and minimal distortion, we mounted a pair of Sony rx100 v cameras to DJI Matrice 100 drone. We calibrated the stereo offset between the cameras before flight, and during flight each camera records a video with 4K resolution at 30 fps.
Data captured in this manner is subject to number of limitations. Compression artifacts in the recorded videos may make features harder to track than when captured RAW format. Because the drone must keep safe distance from the tree, complete 360 coverage of given branch is often infeasible. This lack of coverage is compounded by occlusions caused by other branches and leaves (in seasons when the latter present). Furthermore, the fact that the tree may be swaying slightly in the wind even on a calm day violates the rigidity assumption upon which many multi-view recon- struction algorithms rely. Since we know from the data collection phase that our data coverage will be incomplete, we will need to rely on procedural generation, inpainting, "hallucinating" structure, etc. in order to complete the model.
After capturing the raw data, we augment it to begin to estimate the 3D structure of the environment. We subsample the videos at sparse 1 or fps and use the Agisoft PhotoScan tool to run structure from motion and multi-view stereo on those images, yielding a set of estimated camera frames and dense point cloud. We align cameras and point clouds from separate structure from motion problems by performing rigid fit sparse set of control points. This is standard workflow also supported by open-souree tools (Wu, 2011; may be poorly aligned (or in Schönberger Frahm, 2016; Moulon et al., 2016). Some cameras cases, severely incorrect that they require manual correction). Once the cameras are relatively close, one can utilize an inverse rendering approach like that of Loper & Black (2014) adjusting the misaligned cameras' parameters relative the point cloud. In the case of more severely misaligned cameras, one may select correspondences between 3D points and points in the misaligned image and then find the camera's 's extrinsies by solving a perspective-n-pointt problem (Fischler & Bolles, 1981).
In the supplemental appendices, describe approach constructing large scale geometry using this processed data. Recovering "medium" scale structures that are not captured in the point cloud, however, problem that lends itself well learning- based treatment.
4 ANNOTATION AND LEARNING
Annotating images is a challenging task for human labelers and automated methods alike. Branches and twigs heavily occlude one another, connectivity can be difficult to infer, and the path of even relatively large branch can often not be traced visually from a single view. Thus it is desirable to augment the image data during annotation to aid human labelers.
Aiisoftt PhotoScan, http:// http:///www.///p// agi soft.. com/
3

---

Under review as a conference paper at ICLR 2021
Figure 2: Human labelers use our annotation tool to draw curves with positions, thicknesses, con- nectivities, and unique identifiers images of the tree.
One method for aiding the labeler is to automatically extract a "flow field" vectors tracking the anisotropy of the branches in image space (see Figure 6). The flow field is overlaid on the image in the annotation tool, and the labeler may select endpoints to be automatically connected using the inojection-advectei
scheme discussed in Section 5.3. Section 5.3 also discusses how generate the flow field itself, after first creating segmentation mask. Note that segmentation (i.e. discerning tree not free for each pixel in the image) is simpler problem than annotation (i.e. discerning medial axes, topology, and thickness in image space).
Obtaining segmentation masks is straightforward under certain conditions, e.g. in areas where branches and twigs are clearly silhouetted against the grass or sky, but segmentation can be dif- ficult visually dense regions of an image. Thus, we explore deep learning- -based approaches for performing semantic segmentation on images from our dataset. In particular, we use U- Net (Ronneberger et al., 2015), state-of-the-ar fully convolutional architecture for segmentation; the strength of this model lies in its many residual connections, which give the model the capacity to retain sharp edges despite its hourglass structure. See Section 5.2 for further discussion.
5 EXPERIMENTS
Since the approach to large scale structure discussed in Appendix A works sufficiently well, focus here on medium scale branches.
5.1 IMAGE ANNOTATION
We present human labeler with interface for drawing piecewise linear curves on overlay of tree image. User annotations consist of vertices with 2D positions image space, per-vertes branch thicknesses, and edges connecting the vertices. Degree-1 vertices are curve endpoints, degree-2 vertices lie on the interior of curve, and degree-3 vertices exist where curves connect. A subset of the annotated vertices are additionally given unique identifiers that are used to match common points between images; these will be referred to as "keypoints" and are typically chosen as bifurcation points or points on the tree that are easy to identify in multiple images. See Figure 2.
We take advantage of our estimated 3D knowledge of the tree's environment in order to aid human labelers and move towards automatic labeling. After some annotations have been created, their corresponding 3D structures are generated and projected back into each image, providing rough visual cues for annotating additional images. Additionally, since we capture stereo information, augment our labeling interface to be aware of stereo pairs: users annotate one image, copy those annotations to the stereo image, and translate the curve endpoints along their corresponding epipolar lines to the correct location in the stereo image. This curve translation constrained to epipolar lines (with additional unconstrained translation necessary to account for error) is much less time consuming than labeling the stereo image from scratch.

---

Under review as a conference paper at ICLR 2021
Figure 3: A set of anisotropic kernels is used to obtrain directional activations in segmentation masks for both perceptual loss and flow field generation.
Human labelers often identify matching branches and twigs across images by performing human optical flow, toggling between adjacent frames of the source video and using the parallax effect to determine branch connectivity. This practice is an obvious candidate for automation, e-g. by annotating an initial frame then automatically carrying the annotated curves through subsequent frames via optical flow. Unfortunately, the features of interest are often extremely small and thin and the image data contains compression artifacts, making automatic optical flow approaches quite difficult. However, it our hope that in future work the same tools that aid human labelers can be applied to automatic approaches making them more effective for image annotation.
5.2 DEEP LEARNING
In order to generate flow fields for assisting the human labeler as discussed in Section 4, we first ob- tain semantic segmentations of tree and not free using deep learning approach. To train network for semantic segmentation, we generate a training dataset by rasterizing the image annotations as bi- nary segmentation masks of the labeled branches. From these 4K masks, then generate dataset of 512 x 512 crops containing more than 4000 images. The crop centers are guaranteed to at least 50 pixels away from another, and each crop guaranteed correspond to segmentation mask containing both binary values. The segmentation problem on the raw 4K images must work on image patches with distinctly different characteristies: the more straightforward case of branches silhouetted against the grass, and the more complex case of highly dense branch regions. Therefore, we split the image patches into two sets via k-means clustering, and train two different models to segment the two different cases. For the same number of training epochs, our two-model approach yields qualitatively better results than the single-mode model approach.
Instead of directly using the standard binary cross entropy loss, the sparseness and incompleteness of our data led us use a weighted variant, in order to penalize false negatives more than false positives. As a further step to induce smoothness and sparsity in our results, we introduce a second order regularizer through the L2 difference of the output and ground truth masks' gradients. We also experiment with auxiliary loss similar to the VGG perceptual loss described in Mosinska et al. (2018), but instead of using arbitrary feature layers of a pretrained network, we look at the L1 difference of hand-crafted multiscale directional activation maps. These activation maps are pro- duced by convolving the segmentation mask with a series of Gabor filter-esque (Jain & Farrokhnia, 1991) feature kernels (k(0, r, or: IR2 [0, [0,..., NJ²). where each kemel is scale-aware and piece- wise sinusoidal (see Figure 3). A given kernel k(0, r, σ) detects branches that are at an angle 0 and have thicknesses within the interval [r, orl. For our experiments, we generate 18 kernels spaced 10 degrees apart and use N = 35, 4, and 1.8.
Figure illustrates two annotated images used in training and the corresponding learned semantic segmentations. Note that areas of the semantic segmentation that are not part of the labeled anno- tation may correspond true branches or may be erroneous; for the time being a human must still choose which pieces of the semantic segmentation to use in adding further annotations.
5.3 LEARNING-ASSISTED ANNOTATION
To generate a flow field, we create directional activation maps as in Section 5.2 again using the kernels from Figure then perform clustering step the resulting per-pixel histograms of gra- dients (Dalal& & Triggs, 2005) to obtain flow vectors. Each pixel labeled as tree with sufficient more principal directions; pixels with more than direction confidence is assigned
5

---

Under review as a conference paper at ICLR 2021
Figure 4: (Left) Image masks generated from image annotations and used training data. (Right) Outputs of the segmentation network.
Figure 5: (Top left) A ground truth mask of the tree taken by flattening the image annotation data into a simple binary mask. (Bottom left) A visualization of flow directions estimated by applying directional filters to the ground truth mask. (Right) Medial axes of the tree branches estimated from the flow field.
potentially branching points. We find the principal directions by detecting clusters each pixel's activation weights; for each cluster, take the sum of all relevant directional slopes weighted by their corresponding activation values.
Having generated flow field of sparse image space vectors, trace approximate medial axes through the image via an alternating projection-advection scheme. From given point on a branch, estimate the thickness of the branch by examining the surrounding flow field and project the point to the estimated center of the branch. We then advect the point through the flow field and repeat this process. In areas with multiple directional activations (e.g. at branch crossings or bifurcations). our advection scheme prefers the direction that deviates least from the previous direction. More details about this scheme may found in the supplemental material. By applying this strategy to flow fields generated from ground truth image segmentations, we are able to recover visually plausible medial axes (see Figure 5). However. medial axes automatically extracted from images without ground truth labels are error prone. Thus, we overlay the flow field on the annotation interface and rely on the human labeler. The labeler may select curve endpoints in areas where the flow field is visually plausible, and these endpoints used to guide the medial axis generation. See Figure 6 for an example flow field generated from the learned segmentation mask and the supplemental material for demonstration of semi-automated medial axis generation.

---

Under review as a conference paper at ICLR 2021
Figure 6: The trained network infers segmentation mask (bottom left) from an input image (top left). We then estimate flow field (right) by applying anisotropic filters to the segmentation mask. A labeler can specify endpoints between which medial axis and thickness values are automatically estimated (right, in green).
5.4 RECOVERING MEDIUM SCALE BRANCHES
Given set of image annotations and camera extrinsics obtained via structure from motion and stereo calibration, we first construct piecewise linear branches in 3D. We triangulate keypoints that have been labeled in multiple images, obtaining 3D positions by solving for the point that minimizes the sum squared distances to the rays originating at each camera's optical center and passing through the camera's 's annotated keypoint. We then transfer the topology of the annotations the 3D model by connecting each pair of 3D keypoints with line segment if curve exists between the corresponding keypoint pair in any image annotation.
Next, we subdivide and perturb the linear segments connecting the 3D keypoints to match the cur- vature of the annotated data. Each segment between two keypoints is subdivided by introducing additional vertices evenly spaced along the length of the segment. For each newly introduced ver- tex, we consider the point that the same fractional length along the image-space curve between the corresponding annotated keypoints in each image for which such curve exists. We trace rays through these intra-curve points to triangulate the position of each new vertex in the same way that triangulated the original keypoints.
Finally, we estimate the thickness of each 3D vertex beginning with the 3D keypoints. We estimate the world space thickness of each keypoint by considering the corresponding thickness in all anno- tated camera frames. For each camera in which the keypoint is labeled, we estimate world space thickness using similar triangles, then average these estimates get the final thickness value. We then set the branch thickness of each of the vertices obtained through subdivision simply by inter- polating between the thicknesses of the keypoints either end of the 3D curve. Using this strategy, we recover a set of 3D positions with local cross-sectional thicknesses connected by edges, which is equivalent the generalized cylinder representation employed in Appendix A.
The human users of our annotation tools encounter the traditional trade-off of stereo vision: it is easy identify common features images with small baseline, but these features triangulate poorly exhibiting potentially extreme variance in the look-at directions of the corresponding cameras. Con- versely, cameras whose look-at directions are close to orthogonal yield more stable triangulations, but common features between such images are more difficult to identify. One heuristic approach is to label each keypoint three times: twice in similar images and once from a more diverse viewpoint. However, it may be the case that some branches are only labeled in two images with small baseline (e.g. a stereo pair). In this case, we propose a clamping strategy based on the topological prior of the tree. Designating "root" vertex of subtree for such annotations, keypoints as usual obtaining noisy positions in the look-at directions of the stereo cameras. We then march from the root vertex the leaf vertices. For each vertex p with location Pz, consider
triangulate the annotated

---

Under review as a conference paper at ICLR 2021
Figure 7: Branches labeled from stereo pair of cameras are visually plausible from the perspective of those cameras (top left), but they can exhibit severe error when viewed from different angle (top right). By clamping these branch positions, one can achieve virtually identical projection to the original cameras (bottom left) while maintaining a nondegenerate albeit "flattened" appearance from different angle (bottom right).
each outhoard child vertex with location Cg- For each camera in which the point is labeled, consider the intersection of the ray from the location of c's annotation to Ca with the plane parallel the image plane that contains Pa; let ce be the intersection point. We then clamp the location of based on a user-specified fraction. This process is repeated
between ç₂ and the original location for each camera in which e is annotated, and we obtain the final location for by averaging the clamped location from each camera. See Figure 7.
CONCLUSION AND FUTURE WORK
We presented an end-to-end pipeline for reconstructing a 3D model of botanical tree from RGB im- age data. Our reconstructed model may be readily simulated create motion in response to external forces, e.g. to model the tree blowing in the wind (see Figure 8). We use generalized cylinders to ini- tialize an articulated rigid body system, noting that one could subdivide these primitives as desired for additional bending degrees of freedom, decrease their resolution for faster performance mobile devices. The simulated bodies drive the motion of the textured triangulated surfaces and/or the point cloud data as desired.
Although we presented one set of strategies to go all the way from the raw data to a simulatable mesh, it is hope that various researchers will choose to apply their expertise and improve upon various stages of this pipeline, yielding progressively better results. In particular, the rich topological information in the annotations has great potential for additional deep learning applica- tions, particularly for topology extraction (Ventura et al., 2018; Máttyus et al., 2017; Xue et al., 2018) and 2D 3D topology generation (Estrada al., 2015).
Figure 8: The tree model is deformed from its rest pose (left) to an exaggerated pose (right) via simulation.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sameer Agarwal, Noah Snavely, lan Simon, Steven M Seitz, and Richard Szeliski. Building rome in day. In Computer Vision, 2009 IEEE 12th Int. Conf. on, pp. 72-79. IEEE, 2009.
Iro Armeni, Ozan Sener, Amir Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1534-15555 1543,-54,,220 2016.
Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene under- standing, arXiv preprint
105, 2017.
Jules Bloomenthal. Modeling the mighty maple. In ACM SIGGRAPH Computer Graphics, volume 19, pp- 305-311, ACM, 1985.
Navnect Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 2005. CVPR 2005, IEEE Computer Society Conference on, volume 1, pp. 886 893.
IEEE, 2005.
Rolando Estrada, Carlo Tomasi, Scott C Schmidler, and Sina Farsiu. Tree topology estimation. IEEE Transac- lions on Puttern Analysis Machine Intelligence, (1):1-1, 2015.
Martin A Fischler and Robert C. Bolles. Random sample consensus: paradigm for model fitting with applica- tions to image analysis and automated cartography. Communications of the ACM, (816))381-335, 1981.
Alvaro Fuentes, Sook Yoon, Sang Chcol Kim, and Dong Sun Park. A robust deep-Icaming- baa-ing-binddd based detector for real-time tomato plant discases and pests recognition. Sensors, 17(9):2022, 2017.
Yasutaka Furukawa and Jean Ponce. Accurate dense, and robust multiview stereopsis. IEEE transactions on pattern analysis and machine intelligence. 22(8):3622-1772) 2010.
Jingwei Huang, Angela Dai, Leonidas Guibas, and Matthias Nießner. 3DLite: towards commodity 3d scan- ning for content creation. ACM Trans. Graph., 36(6) 2017.
Anil K Jain and Farshid Farrokhnia. Unsupervised texture texture segmentation using gabor filters. Pattern recognition,
24(12):1 167-1186 1991.
Yifeng Jiang and C Karen Liu. Data-augmented contact model for rigid body simulation. arXiv preprint arXiv: 04019, 2018
Justin Johnson, Alexandre Alahi, and Li Fei-Fci. Perceptual losses for real-time style transfer and super- resolution. In European Conference Computer Vision, pp. 694-711. Springer, 2016.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning cntegory-speciffe mesh reconstruction from image collections. In The Eurppaa Conf. Comput. Vision (ECCV), pp. 371-386, September 2018.
Alina Kloss, Stefan Schaal, and Jeannette Bohg. Combining learned and analytical models for predicting action effects. arXiv preprint 170.00171, 2017.
Guosheng Lin. Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi path refinement networks for high- resolution semantic segmentation. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
Aristid Lindenmayer. Mathematical models cellular interactions development filaments with one-sided inputs. Journal of theoretical biology, 18(3):280-299, 1968.
Yotam Livny, Feilong Yan, Matt Olson, Baoquan Chen, Hao Zhang, and Jihad El-sana. Automatic reconstruc- tion of tree skeletal structures from point clouds. Proc. SIGGRAPH Asia 2010, 29:151:1-151:8, 2010.
Matthew M Loper and Michael J Black. OpenDR: An approximate differreniable renderer. In European Conf. on Comput. Vision, pp. 154-169, Springer, 2014.
Gellért Máttyss Wenjie Luo, and Raquel Urtasun. Deeproadmapper: Extracting road topology from aerial images. In The IEEE International Conference on Computer Vision (ICCV), 2017.
Agata Mosinska, Pablo Mrquez-Neila, Mateusz Koziski, and Pascal Fua. Beyond the pixel-wise loss for topology-aware delineation, In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018

---

Under review as a conference paper at ICLR 2021
Pierre Moulon, Pascal Monasse, Romuald Perrot, and Renaud Marlet, Openmvg: Open multiple view geometry. In International Workshop on Reproducible Research Pattern Recognition, pp. 60-74. Springer, 2016.
Xue Bin Peng. Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Trans. Graph., 36(4):41, 2017.
Przemyslaw Prusinkiewicz. Mark Hammel, Jim Hanan, and Radomir Mëch. Visual madels of plant develop- ment. Handbook of formal languages, pp. 535 597. Springer, 1997.
Guoxiang Qu Wenwei Zhang, Zhe Wang, Xing Dai, Jianping Shi, Junjun He, Fei Li, Xiulan Zhang, and Yu Qiao. Stripnet: Towards topology consistent strip structure segmentation. In Proceedings of 26th ACM International Conference on Multimedia, MM '18 pp. 283-291, New York, NY, USA, 2018. ACM. ISBN 978-1-4500 1681--403-5565-7, doi: 10. 10.1145/3240508.223 32405533. 3240508 3240553.
URL http: /doi. acm. org/10..///..oo /ttg//1111///////
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
Johannes L Schönberger and Jan Man-Michacl Frahm, Structure- frommmotoonn revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104-4113, 2016.
Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. comparison and evaluation multi-view stereo reconstruction algorithms. Proc. of the IEEE Conf. on Comput. Vision and Pattern Recognition (CVPR), pp. 519-528. IEEE, 2006.
Ondrej Stava, Sören Pirk, Julian Kratt, Baoquan Chen, Radomír M&ch, Oliver Deussen, and Bedrich Benes. Inverse procedural modelling of trees. Computer Graphics Forum, volume 33, pp. 118-131. Wiley Online Library, 2014.
Ping Tan, Gang Zeng, Jingdong Wang. Sing Bing Kang, and Long Quan. Image-based tree modeling. In ACM Trans. Graph., volume 26, pp. 87. ACM, 2007.
Carles Ventura, Jordi Pont-Tusct, Sergi Caelles, Dovis-Kokitsi Maninis, and Lue Van Gool. Iterative deep learning for road topology extraction. In Proc. of the British Machine Vision Conf. (BMVC), 2018.
Jason Weber and Joseph Penn. Creation and rendering realistic trees. In Proc. 22nd Ann. Conf. Comput. Graph. Int. Tech., pp. 119-128. ACM, 1995.
Changchang Wu. VisualSFM: A visual structure from motion system. http: http:://ccw...e me fm/, f0/,0011..
Changchang Towards linear-time incremental structure from motion. Vision-3DV 2013, 2013 International Conference on, pp. 127-134. IEFF, 2013.
Ke Xie, Feilong Yan, Andrei Sharf, Oliver Deussen, Baoquan Chen, and Hui Huang. Tree modeling with real tree-parts examples. IEEE TVCG, 22(12):2008 2618, Dec 2015.
Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans-Peter Seidel, and Christian Thcobalt, MonoPeriCap: Human performance capture from monocular video. ACM Transactions on Graphics (TOG), 37(2):27, 2018
Tianfan Xue, Jiajun Wu. Zhoutong Zhang, Chengkai Zhang. Joshua B. Tenenbaum, and William Freeman. Seeing tree structure from vibration. In The European Conf. on Comput. Vision (ECCV), pp. 748-764 September 2018.
Ying Zheng, Steve Gu, Herbert Edelsbrunner, Carlo Tomasi, and Philip Benfey. Detailed reconstruction of 3d plant root shape. In Comput. Vision (ICCV), 2011 IEEE Int. Conf. on, pp. 2026-2033 IEEE, 2011.
Jin Zhou. Ananya Das. Feng Li, and Baoxin Li. Circular generalized cylinder fitting for 3d reconstruction in en- doscopic imaging based on MRF. In Comput. Vision and Pattern Recognition Workshops, 2008, CVPRW'08. IEEE Comput. Society Conf. on, pp. 1-8. IEEE, 2008.
Silvia Zuffi, Angjoo Kanazawa, and Michael J Black. Lions and tigers and bears: Capturing non-rigid, 3d. articulated shape from images. In Proc. of the IEEE Conf. Comput. Vision and Pattern Recognition (CVPR), pp. 3955-3963, 2018.
10


Reviews:
[{"review_id": "tcL0clZKBJu", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of geometrical and topological 3D reconstruction of a (botanical) tree using a drone-mounted stereo vision system and deep learning-based/aided tree branch image annotation procedures.   This is an interesting computer vision 3D reconstruction task, which has important practical applications (e.g in AR/VR or for plant phonemics study), however has not been extensively researched in the past.   Part of the reasons are due to some unique challenges that the problem of  tree reconstruction is facing, in particular, how to accurately recover complex visual occlusions caused by dense tree branches and leaves, and how to ensure the reconstructed topology is accurate.   \n\nThe paper presents a working system, a pipeline, that the authors have developed.  Instead of describing the procedures for recovering  3D geometry reconstruction of the tree, the authors focused their attention in this paper on the recovery of (medium scale) tree branches structure , such as  branch semantic segmentation, tree truck and brach width and direction estimation, as well as the topology reasoning.     They approach such medium-scale reasoning task through a combined used of supervised learning, and semi-automated manual labelling.    Specifically, a binary semantic-segmentation task to separate object from background (i.e.  tree from non-tree) is conducted via a U-Net type neural networks,  followed by a tree branch orientation) estimation  by applying a hand-crafted anisotropic orientation filter banks.   These orientation filters banks have a  Gabor-like steerable filter shape.  Outputs (or activations) of such orientation filters, i.e. the estimated  flow field,  are used to infer the dominating directions of the tree branch in question.  Finally, other geometric parameters (such as branch width, median axis, and cluster membership) are estimated to conclude the medium-scale structure recovery task.  \n\nThe method is tested on a limited number of real images, and impressive results are obtained.  \n\nPluses:  \n- interest and important problem, not been extensively investigated in the past. \n- impressive reconstruction results, as demonstrated in the companioning video. \n- standard reconstruction pipeline, easy to follow. \n\nMinus: \n -  Ad hoc, heavy-handed labelling  pipeline;  strong in engineering, but weak in original scientific contribution.  The paper feels more like an experimental report suitable for ICRA/IROS, and less of a scientific paper in the field of Representation Learning or ICLR. \n- limited theoretical or methodological contribution.  Reading the paper won't learn much novel insights in solving similar tasks. \n-  Insufficient experiments. It seems only one instance of test is reported/presented in the paper and in the supp. material.\n-  The entire labelling process is not an end-to-end solution: rather it requires constant human intervention.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "bWGXZe8yGzR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers an interesting and challenging problem, namely reconstructing 3D structures with very fine details from image data captured \"in-the-wild\", i.e., taken in uncontrolled environments. More concretely, the paper considers the problem of obtaining a detailed 3D reconstruction of trees that results in a 3D model that can be animated.\n\nThe paper proposes a sound pipeline suitable for the task at hand: first, images of the tree are captured from cameras mounted on a drone to allow capturing the tree from a wide range of angles. Structure-from-Motion is then used to recover the larger structures in the tree from these images. Next, finer structured are recovered. An annotator first annotates branches in some of the images. The known 3D structure is thereby used to aid with the annotation task by using the known epipolar geometry. These labels are then used to train two U-Net architectures (one to classify branches in front of a grassy ground and the other to handle denser regions with multiple branches). Applying these networks on the images then provides initial segmentations that are used to aid the annotator. In particular, a flow field is estimated to propagate information about branches and their thickness. Given the annotation, the corresponding 3D structures are then estimated using triangulation, using a form a smoothness prior on the branch geometry.\n\nThe proposed approach is sound and seems suited for the task at hand. However, I am not fully convinced that one would want to apply it in practice for multiple reasons:\n1) As stated in the introduction, using general purpose 3D reconstruction approaches can lead to sub-optimal results when constructing harder structures such as trees. Thus, the paper starts from a general purpose method (Structure-from-Motion) and adds processing steps designed to recover detailed tree branches. However, it is unclear how it compares against non-general purpose 3D reconstruction methods. For example, [Häne et al., Joint 3D scene reconstruction and class segmentation, CVPR 2013] and [Häne et al., Class Specific 3D Object Shape Priors Using Surface Normals, CVPR 2014] show that using class-specific priors can significantly improve reconstruction quality.\nThere is work on reconstructing fine details based on connectivity information, e.g., [Stühmer et al., Tree Shape Priors with Connectivity Constraints using Convex Relaxation on General Graphs, ICCV 2013] and [Oswald et al., Generalized Connectivity Constraints for Spatio-temporal 3D Reconstruction, ECCV 2014]. These approaches seem to be very suitable for the task at hand. Unfortunately, these methods are not even discussed in the paper.\n[Yücher et al., Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction, ACM TOG 2016] and [Yücher et al., Depth from Gradients in Dense Light Fields for Object Reconstruction, 3DV 2016] show impressive results for reconstructing fine details from light-fields (e.g., obtained via densely sampled videos), including recovering small branches of vegetation. These methods also seem to be applicable to the problem at hand. Unlike the proposed approach, they do not seem to require manual annotation.\nNone of these methods, which seem suitable alternatives for the proposed approach, are discussed in the paper or compared against.\n2) The importance of the individual steps of the proposed pipeline is unclear to me:\n  a) How much does training the networks for segmentation help with the workflow, i.e., by using a network to aid the annotator, how much more human time is saved? Ideally, the paper would provide run-times for the individual stages to show the importance of this task.\n  b) How important is it to train two segmentation networks instead of a single one? Unfortunately, the paper does not provide an ablation study here.\n  c) How does the proposed approach of using learning compare to advanced annotation tools such as [Acuna et al., Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++, CVPR 2018]?\n3) It is unclear to me how general the proposed approach is. Can it be readily be applied to other types of trees? Or would it require significant changes? Unfortunately, the paper only provides results for a single tree of a single species.\n\nIn my opinion, the presentation of the proposed method can be significantly improved:\n1) Given the title and abstract, I would have expected the paper to discuss how simulatable geometry is estimated. In addition, the need for simulatable geometry features prominently in the introduction. However, this part is never really explained in the paper and the paper mostly refers to the supplementary material.\n2) Given that the paper is about reconstructing trees, I would have expected that it shows some qualitative or quantitative results for the reconstructed trees. Qualitative results are shown in the supplementary material, but not in the paper.\n3) In general, the paper is rather light on details. I had the impression that the supplementary material (both the pdf and the video) provide a better explanation of the proposed method than the main paper (at least watching the video helped me to better understand how the proposed approach is structured and how the individual stages depend on each other). It would be good to move some of these details and descriptions from the supplementary material to the paper.\n4) I find the name of the \"Experiments\" section a bit misleading as it mostly discusses parts of the method and does not show much experimental results.\n5) It is unclear to me why one would not simply use a hardware-synchronized stereo camera instead of two individual cameras. Hardware synchronization would remove problems such as slight motion of the tree between the two images. One could even use a multi-camera system such as the one from [Gohl et al., Omnidirectional visual obstacle detection using embedded FPGA, IROS 2015] to be able to see a larger part of the tree.\n\nOverall, I think the paper considers an interesting problem and proposes a sensible solution. However, the proposed method currently stands rather isolated, both compared to previous work applicable in this setting and in terms of exploring alternatives for some of its components / showing the importance of its components. Together with problems in the presentation, I feel that the paper in its current form is not yet ready for publication and would require a significant revision (improved presentation, comparisons with existing work, more detailed experimental evaluation).\n\n### After rebuttal phase ###\nSince the authors did not provide a revised version of the paper or addressed my comments in detail, I do not see a reason to change my initial recommendation. I thus recommend to reject the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "TQ9Oc95RA6d", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the interesting question of how to simulate 3D shape of trees: given sets of 2D input images taken from drone cameras, the process consists of building point clouds, and textured triangle meshes, and creating a skinned cylindrical articulated rigid-body 3D shape. It is an important problem in artificial life, and has been investigated in the past, by e.g. L-system. In this work, a systematic pipeline is proposed and reasonable 3D shape result is demonstrated. I also enjoy the demo video and appreciate the amount of efforts devoted to annotate the tree data manually.\nOne the other hand, there are two main issues that prevent it from being considered in this conference venue. The first concerns the sparse result. With only one 3D shape being presented, it is difficult to understand the capacity of the proposed method; second, no or very weak machine learning and related technical contribution exist in the paper. Most of the text is devoted to 3D graphics and computer vision pipelines. Overall it seems unfit to ICLR audience.  \nThere are a few other comments that I hope would be useful for the authors:\n1. lack of technical contributions. Currently every module in the pipeline is off-the-shelf. The authors are encouraged to conconsider developing a new learning based method to tackle (maybe part of) the problem. The could dramatically change the feel of the current work. \n2. more empirical studies. The current dataset seems to consist of only one single tree. It would be much more convincing to show multiple trees acquired at very different location/season/species, etc.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "42s6luywKnK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors describe a processing pipeline for producing 3D models of trees from stereo visual data, which is later extended with a DNN for segmentation.\n\nThe paper is more on the practical, or application side, although very interesting especially for tackling a very complex task, which was bound to present challenges since segmenting thing and long structures is often hard, as the authors mention.\n\nThis kind of paper is often challenging to review, because it is clear that a lot of good work has been done, and the overarching goal of the project seems interesting. It is hard to assess the paper, though, if apart from reporting on the methods the results are only final models obtained. It might suffice, although the authors should be strongly encouraged to find some kind of experiment that could have been done to validate the results more. For example, there could be objective assessments of the models to see if they fit some kind of physical prediction about the trees shape and laws of physics. Or perhaps the labeling procedure could have been investigated, and the segmentation shown to be accurate and to accelerate the labeling. Even a small result, as long as there is some kind of objective analysis that can be fit in a hypothesis-testing framework, can go a long way. How do I know these models produced are any good, because I looked at the paper and found them nice? It is very important when writing a scientific paper to make sure some kind of objective analysis can be offered.\n\nIt is also important to make the proposal very clear from the beginning. The title, abstract and first section of the paper focus too much on the long term intentions of the authors, instead of objectively and concisely describing exactly what pertains to the paper. The authors should try to fit in the abstract a short description of the processing pipeline that explains that stereo visual data was used to label the tree images, then the labels used to train a U-net with Gabor filters, and this used again to speed up labeling, for example. Whatever is the gist of it, this should be soon in the abstract and everywhere in the paper. Avoid making a mystery of what is the work done, on the contrary, repeat many times what has been done, completely, in different levels of detail.\n\nThe recommendation is to accept the paper mostly by virtue of the authors approaching an interesting problem and having produced some results. It's only marginal, though, because the papers still feels a little incomplete an perhaps lacks novelty.\n\nSome specific remarks:\n\nWhat makes the model \"simulable\"? How do we know it was achieved? Or was it a constraint from the beginning?\n\nWhy call it \"Gabor filter-esque\", how is it not exactly a Gabor filter bank? And a quick note about the filters, have you considered also using quadrature filters, with odd symmetry? The even-symmetry filters used may be more adequate for detecting ridges than edges, it might be nice to have both, as proposed in many papers in edge-detection literature.\n\nUse of classic function bases and filter banks with deep learning is a very interesting, and maybe one of the most interesting aspects of this paper. Perhaps cite other examples of research where this has been done, one example that comes to mind is https://arxiv.org/pdf/1801.10130.pdf\n\nIt seems the labels were only 2D, always, although 3D information was used to greatly help in the process. Why not actually label as 3D?\n\nThe neural network was only employed in 2D, even though much 3D data is available. Could the model have benefited from taking stereo pairs as input, perhaps?\n\nOne very interesting aspect of 3D reconstruction is that sometimes the geometry of objects in 2D is more complicated than in 3D, because we are looking at data distorted by perspective, etc. It would be very interesting to see how this work could benefit from enforcing priors in 3D. Right now the approach still seems to be mostly treating the problem as 2D. What potential do the authors see in leveraging more 3D information?\n\nThe model is said to be based on cylinders, although if the thickness is changing along the segments, is it not more accurately cones?\n\nOne very important topological constraint of the peculiar objects studied is that they appear to create graphs that contain no cycles (!) might be good to mention this interesting fact somewhere in the paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper proposes and demonstrates a method to reconstruct 3d shape for a tree, from drone data.  While the reviewers all appreciated to work, all felt there were many shortcomings of the paper with respect to an ICLR audience:
(a)  no machine learning novelty
(b)  highly interactive data processing method
(c) only one example processed tree shown
(d) inadequate connections with relevant literature on 3d reconstruction, both general purpose, and examples applied to vegetation.
(e) incomplete presentation of the method:   no ablation studies, no listing of the times required for individual steps of the processing.

In view of these concerns, we have decided to reject the paper.  But we hope you find the reviewers' comments helpful, and make use of them in a revision of the work.

Decision: Reject