Probabilistic Plant Modeling via Multi-View Image-to-Image Translation
Takahiro Isokane1,∗

Fumio Okura1,2,∗ Ayaka Ide1 Yasuyuki Matsushita1
1
2
Osaka University
JST PRESTO{isokane,okura,ide,yagi}@am.sanken.osaka-u.ac.jp

Yasushi Yagi1

yasumat@ist.osaka-u.ac.jp

Figure 1: Result of branch structure estimation from simulated and real leafy plant images. From multi-view plant images,
our method infers the branch structure in a probabilistic framework. An explicit graph structure (red lines) can be derived
from the probabilistic plant model.

Abstract
This paper describes a method for inferring threedimensional (3D) plant branch structures that are hidden
under leaves from multi-view observations. Unlike previous geometric approaches that heavily rely on the visibilityof the branches or use parametric branching models, ourmethod makes statistical inferences of branch structures ina probabilistic framework. By inferring the probability of
branch existence using a Bayesian extension of image-toimage translation applied to each of multi-view images, ourmethod generates a probabilistic plant 3D model, which
represents the 3D branching pattern that cannot be directly
observed. Experiments demonstrate the usefulness of the
proposed approach in generating convincing branch structures in comparison to prior approaches.

1. Introduction
We propose an approach to estimate branch (skeleton) structures of plants from multi-view images that are
severely occluded by leaves. Unveiling hidden skeleton
structures is one of the most challenging tasks in computer
vision, because it naturally involves inference of the unobserved structures. Skeleton estimation is actively studied forhuman pose estimation [6]. A recent trend in image-based
human skeleton estimation relies on the prior knowledge of∗ Authors contributed equally.

the relationship among joints for extracting and interpreting joints from images. Unlike human bodies, the branch
structures of plants are less organized, namely, the numberof joints and their connections are generally unknown. In
addition, heavy occlusions due to leaves makes the problem
harder. These aspects pose a unique challenge in plant’s
structure estimation. Although difficult, branch structure
estimation has a number of applications, such as synthetic
plant generation in computer graphics and computational
agriculture.
Toward this goal, this paper presents a multi-view imageto-image translation approach to 3D branch structure estimation. To estimate the branching paths hidden underleaves, we cast the estimation problem to an image-to-imagetranslation problem [12], which converts an image from one
domain to another domain. In our context, we convert an
input image of a leafy plant to a map that represents thebranch structure, i.e., each pixel containing the predictionof “branch” or “not-branch”. To deal with the uncertainty ofthe prediction, we develop a Bayesian extension of imageto-image translation applied to each of the multi-view images. It yields the prediction together with its reliability,
resulting in probabilistic estimates in a similar manner to a
Bayesian semantic segmentation approach [13].
The probabilistic representation is advantageous in a few
important aspects; not only that we have access to the credibility of the estimates, but also that it allows us to consolidate the view-dependent inferences in a 3D space in awell-defined probabilistic framework. Thus, in our method,2906

instead of directly predicting 3D branch structures, it makes
inference in individual views of the multi-view input andsubsequently aggregate them in a 3D voxel space to obtain aprobabilistic 3D plant model. We also develop a method forgenerating explicit branch structures from the probabilistic
model based on particle flows so that the probabilistic representation can be converted to a form that could be used
in intended applications. A few examples of the output are
shown in Fig. 1.
The primary contributions of this study are twofold.
First, we propose a method of recovering 3D structures of aplant based on Bayesian image-to-image (leafy- to branchimage) translation applied in a multi-view manner. By representing the branchness in a probabilistic framework, weshow that both faithful estimates and comprehensive aggregation over multi-view predictions can be achieved. Second, we show that the branch structure estimation is madepossible by the proposed approach, together with a method
for extracting explicit branch structures from the probabilistic representation. Experimental results show that the proposed method can generate convincing branch structureseven with severe occlusions by leaves and other branches.
In comparison to a traditional tree reconstruction approach,
we found notable improvement in the resultant 3D branching structure.

2. Related Works
Our goal is to reconstruct a 3D branch structure of a plantfrom multi-view images that exhibit severe occlusions due
to plant leaves. At the heart of the proposed method, we develop a Bayesian extension of image-to-image translation
for making statistical predictions of branch existence evenfor the parts that are not observed at all. In what follows,
we discuss the prior arts for 3D reconstruction of plants andtrees, image-to-image translation and Bayesian neural networks that are related to our work.
3D reconstruction of plants and trees. (Semi-) automatic 3D modeling of plants and trees is actively studied
in the graphics community [32] because of their importance as a rendering subject and that their manual modeling is notably time-consuming. Interactive 3D modelingmethods of trees using manually provided hints, e.g., lines,
are proposed in the early 00’s [4, 21]. Growth models oftrees, which include branching rules, are occasionally utilized [10, 29, 22], some of them are augmented by realistic textures [16]. These approaches generally heavily relyon the branch models (or rules), and the resulting structure
cannot deviate much from the presumed models.
On the other hands, an approach based on observations of real-world trees that uses photographs [27] or 3D
scans [34, 17] is shown promising for automatic tree modeling. Several tree modeling approaches using multi-view
images have been proposed [25, 31, 20]. These approaches

have been further extended to single-image based methods [30, 2] for better applicability. The major focus of themost image-based modeling approaches has been to generate 3D tree models that well fit the silhouette or volumes, not necessarily aiming at recovering branch structures. Using multi-view images of bare trees, previous
approaches achieve geometric reconstruction of a branch
structure [18, 28, 35]; however, leaves of plants or trees
make 3D reconstruction considerably difficult due to occlusions. The scope of plant modeling now goes beyond computer graphics and is becoming an important application ofcomputer vision, i.e., analysis of plant shapes and growthfor vision-assisted cultivation and plant phenotyping. Our
method aims to recover branch structures via a multi-view
approach with an emphasis on dealing with uncertainty due
to severe occlusions by leaves.
Apart from the plant and tree context but somehow related, recently human skeleton (structure) estimation showsa great success [23, 6]. The task has a similarity to what westudy in this paper; however, the plant and tree branch structure estimation has its unique difficulty in that the numberof joints and their relationships are generally unknown andcomplex.
Image-to-image translation. Image-to-image translation
aims at transferring contextual or physical variation between the source and target images. The early works on
image-to-image translation include image analogies [11]and texture transfer [8]. Commonly, these approaches divide the image into small patches and transfer the change
based on patch-wise correspondences [14, 3, 7]. More recent image-to-image translation largely benefits from deeplearning, such as convolutional neural networks (CNNs)
with encoder-decoder architectures or using Generative Adversarial Networks (GANs). Pix2Pix [12], which uses conditional GAN, shows impressive performance on a wide
variety of translation tasks. Along this context, Cycle
GAN [37] has shown the possibility of image-to-imagetranslation without paired training images.
Bayesian neural networks To obtain reliability of inference, a Bayesian framework has been used together withneural networks in Bayesian neural networks (BNNs) [19].
Recent BNNs use dropout connections in forward passes
in the prediction stage [9], which yields variations in inference. The estimate is thus given in the form of distribution
rather than a point, effectively modeling the uncertainty ofthe prediction. This approach can be regarded as an approximation of Monte Carlo integration in traditional Bayesian
approaches [15]. An impressive result has been shown inBayesian SegNet [13] that achieves semantic segmentation
augmented by the prediction reliability. Our method alsouses the Bayesian approach in image-to-image translation
task in order for obtaining the probabilistic representationof the branch structures.

2907

Input:Multi-view
plant image

1) Bayesian
image-to-imagetranslation

2) Probabilistic 3D branch
structure generation

3) Explicit branch generation
using particle flows۴ௗ۴௖

Output:3D branch path(tree graph)

Particle…

…

۴௥Figure 2: Overview of our plant’s branch structure modeling. The proposed method first estimates the 2D branch existence
probability on multi-view images using a Bayesian extension of image-to-image translation. The 2D probabilities are thenconsolidated in a 3D voxel space to form a volume of 3D branch probability. An explicit 3D branch structure is generated byparticle flow simulation based on the 3D probability map.
superposition

3. Probabilistic Branch Structure Modeling
Our method takes as input multi-view images of a plantand generates a probabilistic 3D branch structure in a 3D
voxel space. Our method begins with estimating a 2D
probabilistic branch existence map in each of the multiview images based on an altered image-to-image translation method. Once the probabilistic branch existence
map is computed for each view, they are merged in a 3D
voxel space using the estimated camera poses based on a
structure-from-motion method [33] to yield a probabilistic
3D branch structure. Finally, an explicit 3D branch structure is generated by a particle flow simulation, which is inspired by a traditional tree modeling approach [20], using
the probabilistic 3D branch structure. Figure 2 illustrates
the whole pipeline of the proposed method. In what follows,
we explain the individual steps of the proposed method.

3.1. Bayesian image-to-image translation
From a leafy plant image, we first estimate a pixel-wise
2D branch existence probability. The major challenge is toinfer branch structure hidden under leaves, which cannot bedirectly observed by a camera, possibly not from any of the
viewpoints. It is here that we adopt a Pix2Pix approach [12]to image-to-image translation as a mean to derive a statistically valid prediction of the existence of branches in themulti-view images. For our context, we train a Pix2Pix network using pairs of a leafy plant and its corresponding label
map describing the branch region.
To obtain a meaningful probabilistic branch existence as
output, we use the original image-to-image translation approach in a Bayesian deep learning framework [9, 19] byMonte Carlo sampling via dropout at the inference stage.
By adopting this fluctuation and repeating the prediction
multiple times, we obtain a probability of the branch existence for each pixel. This strategy is implemented by inserting dropout layers before the middle four convolution lay-variance(a) Two samples generated by(b) Variance in 100 samples
Bayesian image-to-image translation

Figure 3: Effect of Bayesian image-to-image translation in
branch generation. Two generated samples contain differences in small branches (a). The variance in a larger number of samples (b) shows the uncertainty in the prediction,
which cannot be obtained from a single inference.
ers in the encoder-decoder network of Pix2Pix. To further
increase variations of inference, dropout is additionally applied to skip connections [26] of Pix2Pix. Figure 3 shows anexample of output variations by this Bayesian branch generation. Multiple inferences yield the degree of uncertainty,
which cannot be obtained by a single inference.
As such, we obtain the mean of variational inferencesfrom the image-to-image translation network. By treating
the variational inferences as stochastic samples, each pixelin the mean inference can be regarded as the probability of
branch existence spanned in [0, 1]. For the i-th image Ii , thebranch probability B2Di : R2 → [0, 1] at a pixel x2D ∈ R2
is written as:
1 X
 B2Di (x2D ) =B̂2Di,v (x2D ),
nv v
(1)


B̂2Di,v = πv (Ii ),where πv denotes the Pix2Pix translation from an image Ii

2908

(a)

(b)

(c)

(d)

Figure 4: Visualizations of branch probability: a) an input image, b) the ground truth branch, c) 2D branch probability
generated by Bayesian image-to-image translation, d) a 2D projection of 3D branch probability via 3D aggregation. Althoughimage-to-image translation (c) infers a few branches as low probability due to heavy occlusions (e.g., in the circle), they arerecovered by 3D probability generation (d) due to votes from other views.
to the corresponding branch existence B̂2Di,v with the v-th
variation of random dropout patterns. The probability map
B2Di for each viewpoint i is then obtained by marginalizing
the individual samples B̂2Di,v over random trials v.

Input

Ground truth

Generated branches

3.2. Probabilistic 3D branch structure generation
Once the view-wise probability maps {B2Di } are obtained, our method estimates a 3D probability map B3D ofthe branch structure defined in the 3D voxel coordinates.
From the multi-view input images, we estimate the camera
poses and intrinsic parameters by a structure-from-motion
method [33]. It yields a set of projection functions {θ} thatmap from the 3D voxel to image coordinates, θ : R3 → R2 .
Using the projections {θ}, the probability of the branch existence B3D at voxel x3D ∈ R3 can be computed as a joint
distribution of {B2Di } by assuming their independence as
B3D (x3D ) =Y

B2Di (θi (x3D )),(2)iin which θi represents a projection from the voxel to thei-th image coordinates. This aggregation process can be
regarded as a back projection, which is used in traditional
computed tomography [5]. While any of the views may notconvey complete information of the branch structure due toheavy occlusions, the aggregation effectively recovers thebranch structure in a probabilistic framework as depicted inFig. 4 (d). To avoid numerical instability, Eq. (2) is computed in the log domain.

3.3. Explicit branch generation using particle flows
The probabilistic 3D branch structure can be converted
to an explicit representation of 3D branch models that canbe used for applications in computer graphics and branch
structure analysis. Inspired by a conventional tree modeling approach [20], we develop a branch structure generation
method using particle flows. Instead of relying on the attractor graph that is computed directly from images in [20],Figure 5: Different branch structures (shown in red andblue) generated from the same branch probability map, bychanging position and the number of initial particles.
our method uses the 3D probability map to regulate the particle movement for generating branch path candidates. Finally, the candidates are consolidated via structure refinement that includes smoothing and simplification. The resulting 3D model is represented by a graph that consists ofnodes and edges that correspond to joints and branches. Using the particle-flow-based method, our method can yield
different plant structures from a single probability map, bychanging position and numbers of initial particles as shownin Fig. 5. This is one of the merits of our probabilistic plantmodeling for graphics applications.
We first generate particles proportionally to the log probability map log B3D for avoiding the peaky distribution in
the canonical domain. Also, the root position r ∈ R3 of theplant is set to the bottommost point that has a high probability of being a part of branches. Under these settings, startingfrom random distribution of particles, the particle positions
are iteratively updated in a manner of flow simulation. At
each step t, the (t+1)-th particle position pt+1 is computedby the following update rules:(
pt+1 ← pt + F(pt ),(3)
F(pt ) = λc Fc (pt ) + λd Fd (pt ) + λr Fr (pt ).
As illustrated in Fig. 6, Fc (pt ) and Fd (pt ) represent nor-

2909

Particle flow ‫ܤ‬ଷ஽ ܵ௥ ‫ܘ‬௧ ǡ ‫ݎ‬

unify

Particle
‫ܘ‬௧

۴௖

۴

۴ௗ

۴௥

Root ‫ܚ‬

Figure 6: Particle flow simulation. At each step, particles
move according to the force computed from the 3D branchprobability. The traces of particles are stored as branch pathcandidates.
(a)

(b)

Figure 7: Branch structure refinement: a) Branch structure
candidates by flow simulation, and b) the final output afterrefinement and simplification.
malized vectors toward and parallel to the stream of branchprobability. Fr (pt ) represents the unit direction from pt to
the root point r of the plant. These directions are linearly
combined with weight factors λc , λd , and λr that are determined empirically. The traces of the particles are recorded
in a tree graph as vertices and edges, and a unification ofparticles is treated as a joint.
While the flow simulation generates a lot of branch candidates as shown in Fig. 7(a), they are simplified and refined
to yield the final structure. The refinement process involves(1) Smoothing: Apply low-pass filter along branch paths.
We simply update the position of each vertex to the masscenter of neighboring vertices, (2) Refinement: Move eachvertex toward the direction of local probability maximum
perpendicular to current branch direction, and (3) Simplification: Unify vertices located close to each other, and delete
subtrees that locate in areas with small probability. We iterate the steps (1)–(3) several times and acquire the final
branches as a tree graph structure as shown in Fig. 7(b).

3.4. Implementation Details
Here we describe some implementation details on theBayesian image-to-image translation and 3D branch generation.

Generator training. For generating the view-wise probability maps {B2Di }, we train the Pix2Pix network using images rendered using 10 synthetic plants. The plant models
are created by changing the parameters of a self-organizing
tree model [22], which is implemented in L-studio [24], anL-system-based plant modeler. For each plant, we render
images viewed from 72 viewpoints, and each image is additionally flipped for data augmentation. As a result, the total
number of images is 10 × 72 × 2 = 1440. Because Pix2Pix
employs PatchGAN [12] that divides an image into patches
for training, we find that this number of images is sufficientfor training the generator πi of Eq. (1).
The 2D branch probability estimation is implemented
by modifying a TensorFlow implementation of Pix2Pix [1].
To realize Bayesian image-to-image translation describedin Sec. 3.1, we modify the original implementation byadding dropout layers to each of four central layers of theencoder and decoder blocks. Since we intend to model theuncertainty of contextual information in deeper layers (i.e.,
branch patterns), we set a larger dropout rate in central layers (0.8 for the innermost layer, 0.7 and 0.6 for the next two,and 0.5 for the outermost layer). Although [13] uses a fixed
dropout rate (0.5), they also report that applying dropout
only for central layers yields better performance for segmentation tasks. We additionally use dropout for all the skip
connections with a fixed dropout rate 0.3. The inference isrepeated 100 times by randomly changing the dropout patterns. A previous work reports tens of iterations are sufficient to obtain the probability distributions [13].
Explicit branch generation. For particle flow simulation
described in Sec. 3.3, we generate approximately 10, 000
particles in the 3D space. Let log B3D (Sr (pt )) as the subset of the log-probability map around the particle location
pt within radius r, we computed Fc (pt ) and Fd (pt ) as themass center and a major axis of log B3D (Sr (pt )). We set
λr as a constant (0.1 in our experiment) but change λc andλd according to the distance dc between pt and mass centerof B3D (Sr (pt )) as
λc =dc(1 − λr ),r

λd = 1 − λr − λc .

(4)

In this manner, λc becomes smaller when the mass center
gets closer to pt . The implementation of the weight control
is based on the previous approach [20].
Resolution and processing time. We used 128×128 [px]images for Pix2Pix, and they were upsampled to 1024 ×
1024 [px] for the 3D aggregation process. With our unoptimized implementation, it took approximately 10 [sec] for
the Pix2Pix with variational inference (100 inferences) pera viewpoint using an NVIDIA GeForce GTX 1080 GPU.
Also, it took 1 [min] for 3D aggregation, and 30 [sec] forparticle simulation on a CPU (3.70GHz, 6 cores).

2910

2D
branch map{B2Di }

Number of cameras
Visible branch region
Whole plant region [20]Image-to-image translation
Bayesian image-to-image translation (proposed)

Geometric error(Euclidean distance)
72
36
12
6
4.20
3.98
3.59
18.05
12.37 18.12 24.75 17.44
1.76
2.26
2.40
14.95
1.69
1.74
2.14
14.53

Structure error(difference in joint numbers)
72
36
12
6
0.6
0.6
1.0
40.4
305.4 117.6 65.4 35.0
0.8
8.2
13.6 88.2
0.4
1.4
7.4
60.2

Table 1: The accuracy of 3D branch structure generated using different settings and number of views (averaged over fivesimulated plants). The geometric error is defined on a relative scale.
Semi-circles (three heights, 36 images)

A quarter circle (side-view, 6 images)

A semi-circle (side-view, 12 images)

Whole hemisphere(72 images)

Visualizationin Figure 10

Figure 8: Camera setting employed in the experiments.

4. Experiments
We conduct experiments using simulated and real-world
plant images and assess the quality of the reconstruction
both quantitatively and qualitatively.

cause our branch recovery method is also up to scale likemost multi-view 3D reconstruction methods.
We define the structure error as the difference in joint
numbers between the generated and the ground truth structures. From a tree graph generated by the explicit branch
generation, we count the number of graph vertices, where
the number of connected edges is three (i.e., one incoming
and two outgoing) or more, for assessing the accuracy of
structure recovery.
Using these error metrics, we assess the method using
four different settings described below:• Visible branch region: As the easiest test case, we
use the ground truth branch regions in multi-view images that are not occluded by leaves. It naturally skips
the image-to-image translation process because the 2D
branch structure map {B2Di } is directly given.
• Whole plant region: In this test case, to compare with
the image-to-image translation strategy in our problem, a whole plant (i.e., branches and leaves) region
is directly used without image-to-image translation for
defining the probability map {B2Di }. This setting isakin to the previous tree modeling method [20], which
generates branches that fit the entire volume of a tree
by a particle flow simulation.

4.1. Simulated plants
Quantitative evaluation. We first describe the experimental result using simulated plant models, which have theground truth branches for quantitative evaluation. The accuracy of generated plants by the proposed method is assessedusing two metrics; geometric and structure errors.
The geometric error evaluates the Euclidean distances
between 3D points in the generated 3D branch structure
and points in the ground truth branches. We use a graphrepresentation of the generated 3D branch structure and theground truth. By sampling 3D points on the graph edges ofboth generated and the ground truth branches, we assess thegeometric error. Let g ∈ G and t ∈ T be generated andtrue 3D branch points, respectively. The geometric error isdefined as a bidirectional Euclidean distance d [36] between
the two point sets written asPP
1
G ||g − NT (g)||T ||t − NG (t)||,+d(G, T ) =
2
|G|
|T |
where NG (x) and NT (x) are functions to acquire the nearest neighbor point to x from point sets G and T , respectively, and |G| and |T | denote the numbers of points in G
and T . The geometric error is defined only up to scale be-• Image-to-image translation: In this case, the 2D
probability map {B2Di } are generated by the original
image-to-image translation [12], where the Bayesian
extension is not employed.
• Bayesian image-to-image translation (Proposed): In
this setting, we use the proposed method, with whichbranch probability maps {B2Di } are generated by the
Bayesian image-to-image translation.
For the evaluation, we use five plant 3D models, where
the branching parameters are different from the training
dataset described in Sec. 3.4. To assess the effect of thevarying number of cameras, we generate branch structuresfrom multi-view images (a) covering a whole hemisphere(72 cameras), (b) camera paths on semi-circles at threeheights (36 cameras), (c) a semi-circle (12 cameras), and(d) a quarter circle (6 cameras), as illustrated in Fig. 8.
Results. Table 1 summarizes the accuracy evaluation of
the four settings with varying number of views, and Fig. 9

2911

(a1)

3D branch probability(a2)

Estimated branches(b) Visible branch region(c) Whole plant region(d) Image-to-imagetranslation(e) Bayesian image-to-imagetranslation (Proposed)

Figure 9: Results using a simulated plant. The leftmost column shows (a1) an input image and (a2) the ground truth branch
structure. The other columns show the 3D branch probability and the resultant branch structures using different settings. In
comparison to other settings, the proposed approach (e) generates accurate and stable branch structure.

(a) Ground truth (b) Visible

branches(a) Full hemisphere(72 images)(b) Semicircle, three heights (c) Semicircle, side view
(36 images)(12 images)

Figure 10: Results when reducing the cameras. The groundtruth branch is shown in bottom left corner.
shows some of the results when using 72 images. The proposed approach generates accurate branch structures acrossall the view settings in geometric error, while the accuracy
degrades in the smaller number of views. For the case of“visible branch regions,” although we extracted the groundtruth branch labels seen from each camera, branches are
occluded by leaves and there are parts that are unobserved
from most of the viewpoints, resulting in greater geometricerrors. The case of “image-to-image translation,” the output
includes branch paths that are not supported by the reliability of inference, instead generated from one sample, which
results in small sub-branches (see Fig. 9(d)). Since the pro-(c) All regions (d) Image-to-imagetranslation(e) Proposed

Figure 11: Results using a plant of a different species thanthe ones used for training.
posed approach averages multiple inferences, the estimatedbranches become more stable and precise compared to the
non-Bayesian approach, thus shows better agreement between the estimated probability map and the branch paths.
Figure 10 compares the results of the proposed method
together with the Bayesian image-to-image translation
along the varying number of views. It can be seen that themore views make the estimation more faithful, while themethod is still able to recover the overall branch structures.
Different species. To study the generalization ability, weapply the proposed method to plant models where their
species (i.e., texture and shape of leaves) are different fromthe simulated plants used for training. Figure 11 shows thatthe proposed method still generates accurate branch structures compared to the other approaches even for different
leaf textures.

2912

48 images

15 images(a) Camera position(b) Input image(c) 2D branch probability(d) 3D branch probability(e) Branch structure by
the proposed method(f) Branch structureusing a whole plant region

Naturalness

Figure 12: Results using real plants. Using input images where the foreground plants are extracted (b), the proposed approach
generates convincing branch structures (e), in comparison to branch structures generated by a previous tree reconstruction
approach [20] (f).
6
5
4
3
2
1
0

Real
Simulated
Whole
plant

Image-to-imagetranslation

Proposed

Figure 13: Subjective evaluation of the branch naturalness.

4.2. Real plants
Now we show the result of the proposed method using
the images of real-world plants. In this experiment, we
use the same trained model for the image-to-image translation that is used in the simulation experiment in Sec. 4.1.
To avoid the unmodelled factors in the experiment, we firstmanually extract the plant regions from images, which canbe alternatively achieved by chroma keying. Figure 12
shows the results including the intermediate outputs in comparison to the method [20]. The proposed method qualitatively yields convincing branch structures, even though theimage-to-image translation network is trained using simulated plants. Compared with a previous tree reconstruction
approach [20], our approach shows its effectiveness in generating the branch structure under leaves.
Since for this experiment we did not have access to theground truth, we conducted a small subjective evaluation by
10 participants to assess the perceptual naturalness of output
plant skeletons. The participants were asked to watch theskeletons generated by each method overlaid on the plant
image and assessed naturalness of the skeletons in a 7-stepLikert scale. Figure 13 summarizes the result. The pro-

posed approach yields the best score among the comparisons, and the trend is consistent with the evaluation of simulated plants.

5. Discussion
We presented a plant modeling approach via image-toimage translation to estimate branch structures of 3D plantsfrom multi-view images, even if the branches are occluded
under leaves. The combination of Bayesian image-to-image(leafy- to branch-image) translation and 3D aggregation
generates the branch existence probability in a 3D voxel
space, resulting in a probabilistic model of a 3D plant structure. We have shown that explicit branch structures can begenerated from the probabilistic representation via particle
flow simulation.
The experimental results using simulated plants showedthat the proposed approach is able to generate the accurate
recovery of the branch structure of a plant compared witha previous tree modeling approach [20]. It has also beenshown that a Bayesian extension of image-to-image translation is effective in obtaining a stable estimate of the branch
structure in the form of probability compared with a nonBayesian one. Qualitatively, the result also applies to thereal-world plants as demonstrated in the experiment.
One of the advantages of our probabilistic approach isthat it can generate a variety of plant branch structures froma single branch probability map, by changing the parameters of particle flow simulation, as shown in Fig. 5, which
may benefit computer graphics applications.

Acknowledgement
This work was supported in part by JST PRESTO Grant
Number JPMJPR17O3.

2913

References[1] Image-to-imagetranslationintensorflow.
https://affinelayer.com/pix2pix/.
[2] O. Argudo, A. Chica, and C. Andujar. Single-picture reconstruction and rendering of trees for plausible vegetation
synthesis. Computers & Graphics, 57:55–67, 2016.
[3] C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman.
PatchMatch: A randomized correspondence algorithm forstructural image editing. ACM Trans. on Graphics, 28(3):24,
2009.
[4] F. Boudon, P. Prusinkiewicz, P. Federl, C. Godin, and R. Karwowski. Interactive design of bonsai tree models. Computer
Graphics Forum, 22(3):591–599, 2003.
[5] R. A. Brooks and G. Di Chiro. Theory of image reconstruction in computed tomography. Radiology, 117(3):561–572,
1975.
[6] Z. Cao, T. Simon, S. Wei, and Y. Sheikh. Realtime multiperson 2D pose estimation using part affinity fields. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR’17), 2017.
[7] S. Darabi, E. Shechtman, C. Barnes, D. B. Goldman, and
P. Sen. Image melding: Combining inconsistent images
using patch-based synthesis. ACM Trans. on Graphics,
31(4):82, 2012.
[8] A. A. Efros and W. T. Freeman. Image quilting for texture
synthesis and transfer. In Proc. SIGGRAPH’01, pages 341–
346, 2001.
[9] Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.
In Proc. 33rd Int’l Conf. on Machine Learning (ICML’16),
pages 1050–1059, 2016.
[10] C. Galbraith, L. Muendermann, and B. Wyvill. Implicit visualization and inverse modeling of growing trees. Computer
Graphics Forum, 23(3):351–360, 2004.
[11] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H.
Salesin. Image analogies. In Proc. SIGGRAPH’01, pages
327–340, 2001.
[12] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-toimage translation with conditional adversarial networks. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’17), 2017.
[13] A. Kendall, V. Badrinarayanan, and R. Cipolla. Bayesian
SegNet: Model uncertainty in deep convolutional encoderdecoder architectures for scene understanding. In Proc. 28th
British Machine Vision Conference (BMVC’17), 2017.
[14] S. Lefebvre and H. Hoppe. Parallel controllable texture synthesis. ACM Trans. on Graphics, 24(3):777–786, 2005.
[15] Y. Li and Y. Gal. Dropout inference in Bayesian neural networks with alpha-divergences. In Proc. 34th Int’l Conf. on
Machine Learning (ICML’17), 2017.
[16] Y. Livny, S. Pirk, Z. Cheng, F. Yan, O. Deussen, D. CohenOr, and B. Chen. Texture-lobes for tree modelling. ACM
Trans. on Graphics, 30(4):53, 2011.
[17] Y. Livny, F. Yan, M. Olson, B. Chen, H. Zhang, and J. ElSana. Automatic reconstruction of tree skeletal structuresfrom point clouds. ACM Trans. on Graphics, 29(6):151,
2010.

[18] L. D. Lopez, Y. Ding, and J. Yu. Modeling complex unfoliaged trees from a sparse set of images. Computer Graphics
Forum, 29(7):2075–2082, 2010.
[19] R. M. Neal. Bayesian Learning for Neural Networks.
Springer Science & Business Media, 2012.
[20] B. Neubert, T. Franken, and O. Deussen. Approximate
image-based tree-modeling using particle flows. ACM Trans.
on Graphics, 26(3):88, 2007.
[21] M. Okabe, S. Owada, and T. Igarashi. Interactive design ofbotanical trees using freehand sketches and example-based
editing. Computer Graphics Forum, 24(3):487–496, 2005.
[22] W. Palubicki, K. Horel, S. Longay, A. Runions, B. Lane,
R. Měch, and P. Prusinkiewicz. Self-organizing tree models for image synthesis. ACM Trans. on Graphics, 28(3):58,
2009.
[23] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, and B. Schiele. DeepCut: Joint subset
partition and labeling for multi person pose estimation. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’16), pages 4929–4937, 2016.
[24] P. Prusinkiewicz, R. Karwowski, R. Měch, and J. Hanan. Lstudio/cpfg: A software system for modeling plants. In Proc.
Int’l Workshop on Applications of Graph Transformations
with Industrial Relevance, pages 457–464, 1999.
[25] A. Reche-Martinez, I. Martin, and G. Drettakis. Volumetricreconstruction and interactive rendering of trees from photographs. ACM Trans. on Graphics, 23(3):720–727, 2004.
[26] O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In Proc.
Int’l Conf. on Medical Image Computing and ComputerAssisted Intervention (MICCAI’15), pages 234–241, 2015.
[27] I. Shlyakhter, S. Teller, M. Rozenoer, and J. Dorsey. Reconstructing 3D tree models from instrumented photographs.
IEEE Computer Graphics and Applications, 21(3):53–61,
2001.
[28] O. Stava, S. Pirk, J. Kratt, B. Chen, R. Měch, O. Deussen,
and B. Benes. Inverse procedural modelling of trees. Computer Graphics Forum, 33(6):118–131, 2014.
[29] L. Streit, P. Federl, and M. C. Sousa. Modelling plant variation through growth. Computer Graphics Forum, 24(3):497–
506, 2005.
[30] P. Tan, T. Fang, J. Xiao, P. Zhao, and L. Quan. Single image
tree modeling. ACM Trans. on Graphics, 27(5):108, 2008.
[31] P. Tan, G. Zeng, J. Wang, S. B. Kang, and L. Quan. Imagebased tree modeling. ACM Trans. on Graphics, 26(3):87,
2007.
[32] K. W. Waite. Modelling natural branching structures. Computer Graphics Forum, 7(2):105–115, 1988.
[33] C. Wu. Towards linear-time incremental structure from motion. In Proc. Int’l Conf. on 3D Vision (3DV’13), pages 127–
134, 2013.
[34] H. Xu, N. Gossett, and B. Chen. Knowledge and heuristicbased modeling of laser-scanned trees. ACM Trans. on
Graphics, 26(4):19, 2007.
[35] D. Zhang, N. Xie, S. Liang, and J. Jia. 3D tree skeletonization from multiple images based on PyrLK optical flow. Pattern Recognition Letters, 2015.

2914

[36] J. Zhu, S. Du, Z. Yuan, Y. Liu, and L. Ma. Robust affine
iterative closest point algorithm with bidirectional distance.
IET Computer Vision, 6(3):252–261, 2012.
[37] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. In Proc. Int’l Conf. on Computer Vision (ICCV’17),2017.

2915